{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Classification\n",
    "\n",
    "**Note: This is still a work in progress.**\n",
    "\n",
    "In the [last lesson](https://github.com/VikParuchuri/zero_to_gpt/blob/master/explanations/dense.ipynb), we trained our neural network on a regression task. The network predicted a number, `tmax_tomorrow`, that can take on any value.  Regression in this case refers to the output of the model, not to linear regression (it's confusing, I know).\n",
    "\n",
    "We can also train our model on a classification task.  In a classification task, we want the network to predict which category something belongs to.  In this lesson, we're going to predict if an object detected by a telescope is a star, a galaxy, or a quasar.\n",
    "\n",
    "We could do this with regression, and have our output be `1`, `2`, or `3` to indicate a star, a galaxy, or a quasar.  There are issues with this, though:\n",
    "\n",
    "- In the forward pass of a neural network, we multiply by the weights to get the output.  Let's say we predict a `1`, but we actually should have predicted a `2`.  We would need to increase the weights.  But a galaxy isn't twice as large as a star.  So the mapping doesn't make sense in the context of the input data.\n",
    "- We can get an intermediate value like `1.5` as a prediction, which is hard to interpret.  We'd ideally want percentages indicating the chance that an example is each of the classes.\n",
    "- The network can output values below `1` or above `3`.\n",
    "\n",
    "Classification allows us to solve this problem - it lets us output the probability that an example belongs to each of the classes (star, galaxy, or quasar).  In classification, most of the neural network works the same - the two main things that change are the activation function and the loss function."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Binary Classification\n",
    "\n",
    "To start with, we'll train the network to perform binary classification.  In binary classification, our target is either a `0`, or a `1`.  A `1` means that there is a 100% chance of the item belonging to the category.\n",
    "\n",
    "We're going to set up our target so that a star is a `1`, and other objects are `0`.  So our network will return something close to `1` if it thinks the telescope detected a star, and close to `0`, otherwise.\n",
    "\n",
    "Let's load in our data:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "# Append the path so we can import the correct library\n",
    "sys.path.append(os.path.abspath(\"../data\"))\n",
    "from csv_data import SkyServerBinaryDatasetWrapper\n",
    "\n",
    "# Load in our data.  The wrapper handles the details of loading and processing the data for us.\n",
    "wrapper = SkyServerBinaryDatasetWrapper(\"cpu\")\n",
    "[train_x, train_y], [valid_x, valid_y], [test_x, test_y] = wrapper.get_flat_datasets()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[ 0.16745842, -0.58492272,  1.03148637, -0.34855938, -0.83728027,\n        -0.94605772, -0.99534154, -0.83806089,  0.21085172, -0.21763043,\n        -0.36973112,  1.03148936,  1.30931064],\n       [ 0.16886159, -0.58311429,  0.05243046, -0.16653251, -0.15415531,\n        -0.08264457, -0.02604308, -0.83806089,  0.21085172, -0.21763043,\n        -0.36984929, -0.63621258, -0.87919741],\n       [ 0.17057433, -0.58347525,  0.92156796,  0.86709322,  0.59315368,\n         0.44120145,  0.31452753, -0.83806089,  0.21085172, -0.21147922,\n        -0.05302706, -0.65633905, -0.60919097],\n       [ 0.17455754, -0.58650069, -1.03063038, -0.81362749, -0.63669227,\n        -0.52660429, -0.43092107, -0.83806089,  0.21085172, -0.20532801,\n        -0.36999261,  1.03148936,  1.30931064],\n       [ 0.17482457, -0.58441247, -1.29023238, -1.17251944, -0.37676237,\n        -0.02510121,  0.15827647, -0.83806089,  0.21085172, -0.20532801,\n        -0.36818949,  1.03148936,  1.30931064]])"
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x[:5]"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see above, we have 13 input variables to our network.  We'll use these 13 variables to predict a single target variable:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "outputs": [
    {
     "data": {
      "text/plain": "array([[0],\n       [0],\n       [1],\n       [0],\n       [0]])"
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y[:5]"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see above, the target is `0` when the object is not a star, and `1` otherwise.  We want our network to output a number between `0` and `1` indicating the probability that the object is a star.  This is called binary classification.  Let's learn how to set our network up to do this."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Sigmoid activation\n",
    "\n",
    "In the previous lesson, we didn't apply an activation function to the output of our network - we just output the last layer directly.  But in a binary classification problem, we want to predict either `0` or `1`.  A neural network can output values outside this range, so we need to constrain the outputs.\n",
    "\n",
    "We can do this using the sigmoid function, which will ensure all outputs are inside the range `0` to `1`.  Mathematically, the sigmoid function is $\\sigma=\\frac{1}{1 + e^{-x}}$.  $e$ is a special number with a value around `2.71`, with the property that the natural log of $e$ equals `1`.  We use this exact formula because of how it interacts with the loss function (we'll see this later).\n",
    "\n",
    "Raising the number $e$ to the power $-x$ will result in a number close to `0` when $x$ is high.  This is because a negative exponent is the product of fractions, like $3 ^ {-2}$ is equal to $1/3 * 1/3$.  When $x$ is large, the sigmoid will output a number very close to $1$, because $e^{-x}$ will be close to `0`, and the denominator will be close to `1`.\n",
    "\n",
    "Raising the number $e$ to the power $-x$ when x is a negative number will make the value of $e^{-x}$ very large.  The negative sign in $e^{-x}$ will undo the negative sign on $x$, so we'll raise $e$ to a large positive power.  This will make the denominator very large, and push the sigmoid close to `0`.\n",
    "\n",
    "You can see an example here:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "outputs": [
    {
     "data": {
      "text/plain": "[<matplotlib.lines.Line2D at 0x1778bec50>]"
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7+ElEQVR4nO3deXxU1f3/8ffMJJkkZANCEhICYRNQVkFicKs1NeLelaJfoRS1+qWtiu1XsQrVtqLWKlZpaa1bFyvaX6u1IBQQVCSyL8q+hy2BEJIJ2SaZOb8/EgYiSciEJHdm8no+HvPIzJl7Zz43l8m8uefcc23GGCMAAACL2K0uAAAAdGyEEQAAYCnCCAAAsBRhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApcKsLqA5vF6vDh8+rNjYWNlsNqvLAQAAzWCMUWlpqVJTU2W3N378IyjCyOHDh5Wenm51GQAAoAUOHDigHj16NPp8UISR2NhYSbUbExcXZ3E1AACgOVwul9LT033f440JijByqmsmLi6OMAIAQJA51xALBrACAABLEUYAAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEv5HUY+/vhj3XTTTUpNTZXNZtO77757znWWLVumiy++WE6nU/369dPrr7/eglIBAEAo8juMlJWVadiwYZo9e3azlt+7d69uuOEGXX311dqwYYPuv/9+3XnnnVq4cKHfxQIAgNDj97Vpxo4dq7FjxzZ7+Tlz5qh37976zW9+I0kaNGiQli9frueff145OTn+vj0AAAgxbX6hvNzcXGVnZ9dry8nJ0f3339/oOlVVVaqqqvI9drlcbVUeACAEGGPk8RrVeI3cHq9qPEbVHm/dzajG463XXuM18nqNvEbyGCOvOf34zPseY3yvfa7nTt03koyRau/V3j+zTvme1xn36y9rznhg6rWb+st8aX3Ve91zLPul2iZf3lvpXaJb8Ns/f20eRvLz85WcnFyvLTk5WS6XSxUVFYqKijprnZkzZ+rxxx9v69IAAO2oxuPVyaoauSpq5KqslquyWqWVNSqtrJGrolrl7hpVVHtU4faqotqjymqPKtye2rYzHlfWeFRZXRc0aryq9tYGjDO/WOG/m4enhm4YaYlp06Zp6tSpvscul0vp6ekWVgQAaEiF26PDJRU6XFyho64qHS+r0vGTbhWedPvuF5W5VVzuVpnb0661Oew2hTtsCrfbFR5mV5jdpnCHXeEOmxz22pvdVnurvV97qft692022e3yLWe31b7umc/57ttql7PZbJIkm02y1dVSe/+M9lNPyOa7b5POuF+/vXY930pnv96Z7afe/4yVv/x6Z7yUT3JcpP+/5FbS5mEkJSVFBQUF9doKCgoUFxfX4FERSXI6nXI6nW1dGgDgHDxeo8PFFdp97KR2HyvT/uNlOlxcocPFlTpcUqHi8mq/XzMq3KG4qDDFRoYrLrLuZ1S4OkU4FBXhUFS4Q5HhdT/rHkeFOxQVYfe1O8McigirDRbhDrvCHDZFOOwKqwsb4Xa77PYGvnERkNo8jGRlZWn+/Pn12hYtWqSsrKy2fmsAQDMZY3TwRIU2Hy7RlsMu7T5Wpt3HTmpvYZmqarxNrtspwqHUhCilxEcqMcaprp0i1CUmQomdnOoaE6GuMU4lRNUGjtjIMIU7mOIK9fkdRk6ePKldu3b5Hu/du1cbNmxQly5d1LNnT02bNk2HDh3Sn//8Z0nSPffco5deekn/93//p+9///v68MMP9fbbb2vevHmttxUAAL+4Kqu1dt8Jrd5XpI0Hi/XFIZdKKho+yhHhsCsjMVp9EmPUu1snpSVEKTUhUqkJUeoeH6W4yLB6XQiAv/wOI2vWrNHVV1/te3xqbMfEiRP1+uuv68iRI8rLy/M937t3b82bN08PPPCAXnjhBfXo0UN/+tOfOK0XANpRubtGubuP65OdhVq5t0jb8l1nDfgMd9h0QXKsBqfGq39yjPp2i1Gfbp3Uo3O0HHR5oA3ZjAn88ccul0vx8fEqKSlRXFyc1eUAQFDIO16uJdsKtHT7MX2257jcX+pu6dU1WpdkdNHIXp01JK02gDjDHBZVi1DU3O/vgDybBgDQMkddlfrPpiN6b+NhbTxQXO+5tIQoXT2wmy7t01WjM7ooycKzJ4AzEUYAIMi5a7xasDlfc1fnKXf3cXnrjnfbbVJm76766sAkXT2wm/p2i2FsBwISYQQAglSBq1JvrszTm6vydKz09KzVF/dM0M3DUnXD0FR1i2WaBAQ+wggABJldR0v14oe7NG/TEdXUHQbpFuvUbaN76psX91DPrtbMogm0FGEEAILE3sIyvbB4h97beNh3JsyoXp01cUyGci5KUUQY83cgOBFGACDAHSqu0POLduhf6w/JU3ckJOeiZP3oq/01OC3e4uqA80cYAYAAVe3x6pXle/XC4p2qqK69rss1A5P0wNcuIIQgpBBGACAArdxzXI+++4V2Hj0pSRqd0UXTrh+oET07W1wZ0PoIIwAQQEoqqvXL/2zRO2sPSpK6dIrQI9cP0jcvTuO0XIQswggABIgVuwv1k7c36nBJpWw2afzonvq/nAFKiI6wujSgTRFGAMBiHq/RrMU79NLSXTKmdpr2574zTCN7dbG6NKBdEEYAwEKFJ6v047+v14rdxyVJ370kXY/deKE6OfnzjI6Df+0AYJGtR1ya/PpqHS6pVFS4Q099c4huGZ5mdVlAuyOMAIAFFm8p0I/fWq9yt0d9EjvpD3eMVP/kWKvLAixBGAGAdvbWqjw98q/P5TXSZf266ne3jVR8dLjVZQGWIYwAQDsxxuh3y3br1wu3S5LGjUrXL78+WOEOpnFHx0YYAYB2YIzRb/5be8aMJE25uq9+cu0A5g4BRBgBgDZnjNGz/92u2Ut3S5J+dv0g3XVlH4urAgIHYQQA2tisxTt9QeSxGy/U5Mt7W1wREFjoqASANvT6p3v1wpKdkggiQGMIIwDQRt7bcEg/f3+LJOmB7AsIIkAjCCMA0AZW7S3ST97ZKEn63pgM/fiafhZXBAQuwggAtLJ9hWX6wV/WqNpjNHZwiqbfeCFnzQBNIIwAQCsqrazW5DdW60R5tYb1iNdz3xkuu50gAjSFMAIArcQYo5+8s1G7j5Wpe3ykXp4wSlERDqvLAgIeYQQAWskfPt6jhZsLFOGw6/f/M1JJcZFWlwQEBcIIALSCz/Yc1zMLtkmSZtx8oYanJ1hbEBBECCMAcJ6Ky926/60N8hrpGxen6bbRPa0uCQgqhBEAOA/GGD30/zYp31WpPomd9ItbBnPmDOAnwggAnIe3Vh/Qws0FCnfY9NvxI9TJyVU2AH8RRgCghQ4UleuX/6mdYfWnOQM0OC3e4oqA4EQYAYAWMMbo4X9uUpnbo0syOuvOy7kKL9BShBEAaIE3V+Xp013HFRlu1zPfGsbEZsB5IIwAgJ8KXJWaOb/2NN6f5gxU78ROFlcEBDfCCAD46Yn3t+hkVY2Gpyfoe2MyrC4HCHqEEQDww9LtRzXv8yNy2G168utD5KB7BjhvhBEAaKbKao9mvLdZkjRpTIYuTI2zuCIgNBBGAKCZ/vTJHuUVlSslLlL3f+0Cq8sBQgZhBACaocBVqd8t2y1Jmnb9QMUwuRnQaggjANAMTy/YpnK3Rxf3TNDNw1KtLgcIKYQRADiHjQeK9c91hyRJM266iGvPAK2MMAIATTDG6KkPaucU+caINA1LT7C2ICAEEUYAoAkf7yxU7p7jigiz68GcAVaXA4QkwggANMLrNXq67qjIhEt7KS0hyuKKgNBEGAGARry/6bC2HHEp1hmm/726n9XlACGLMAIADajxePXC4p2SpLuv7KMunSIsrggIXYQRAGjA+5sOa09hmRKiwzXp8t5WlwOENMIIAHyJx2v04pJdkqS7rujDBGdAGyOMAMCXvL/x9FGRiVyVF2hzhBEAOIPXa/Tih7VjRTgqArQPwggAnOG/Wwq0+1iZ4iLDOCoCtBPCCADUMcbo9x/VXgxvQlYGR0WAdkIYAYA6uXuOa+OBYjnD7PreZRlWlwN0GIQRAKgz56M9kqTvjEpXYozT4mqAjoMwAgCSthx26eMdx+Sw23T3lX2sLgfoUAgjACDp1U/3SpKuG5yi9C7RFlcDdCwtCiOzZ89WRkaGIiMjlZmZqVWrVjW5/KxZszRgwABFRUUpPT1dDzzwgCorK1tUMAC0tmOlVfr3hsOSpMnMtgq0O7/DyNy5czV16lTNmDFD69at07Bhw5STk6OjR482uPybb76phx9+WDNmzNDWrVv1yiuvaO7cuXrkkUfOu3gAaA1/W7lfbo9XI3om6OKena0uB+hw/A4jzz33nO666y5NmjRJF154oebMmaPo6Gi9+uqrDS6/YsUKXXbZZbrtttuUkZGha6+9VuPHjz/n0RQAaA9VNR799bP9kqTvX8ZREcAKfoURt9uttWvXKjs7+/QL2O3Kzs5Wbm5ug+uMGTNGa9eu9YWPPXv2aP78+br++uvPo2wAaB3vbzyiwpNudY+P1HWDU6wuB+iQ/JrRp7CwUB6PR8nJyfXak5OTtW3btgbXue2221RYWKjLL79cxhjV1NTonnvuabKbpqqqSlVVVb7HLpfLnzIBoNn+UndU5H8u7aVwB2P6ASu0+Sdv2bJlevLJJ/W73/1O69at0z//+U/NmzdPv/jFLxpdZ+bMmYqPj/fd0tPT27pMAB3QpoPF2nigWBEOu8Zdwt8ZwCp+HRlJTEyUw+FQQUFBvfaCggKlpDR8ePOxxx7THXfcoTvvvFOSNGTIEJWVlenuu+/Wz372M9ntZ+ehadOmaerUqb7HLpeLQAKg1Z0aKzJ2SAqTnAEW8uvISEREhEaOHKklS5b42rxer5YsWaKsrKwG1ykvLz8rcDgcDkm114FoiNPpVFxcXL0bALSmkvJq/Xtj7em8d1zay+JqgI7N76tATZ06VRMnTtSoUaM0evRozZo1S2VlZZo0aZIkacKECUpLS9PMmTMlSTfddJOee+45jRgxQpmZmdq1a5cee+wx3XTTTb5QAgDt7R/rDqqy2quBKbEa2YvTeQEr+R1Gxo0bp2PHjmn69OnKz8/X8OHDtWDBAt+g1ry8vHpHQh599FHZbDY9+uijOnTokLp166abbrpJv/rVr1pvKwDAD8YY/X1VniTp9kt7yWazWVwR0LHZTGN9JQHE5XIpPj5eJSUldNkAOG9r9xfpm7/PVVS4Q6t+do1iI8OtLgkISc39/uY8NgAdzlurDkiSbhjanSACBADCCIAOpbSyWv/ZdESS9F1O5wUCAmEEQIfyn01HVFHtUd9unRi4CgQIwgiADuWt1bVdNN+9pCcDV4EAQRgB0GHsOlqqjQeKFWa36esXp1ldDoA6hBEAHca762snObvqgm7MuAoEEMIIgA7B6zV6d8MhSeKoCBBgCCMAOoS1eSd08ESFYpxhyh6UfO4VALQbwgiADuGf62qPiowdnKLIcC5FAQQSwgiAkFdV49G8TbXjRb4+gi4aINAQRgCEvKXbjslVWaOUuEhl9ulqdTkAvoQwAiDkvbu+tovmluGpctiZWwQINIQRACGtpLxaH247Kkm6lS4aICARRgCEtPlfHJHb49XAlFgN6s5Vv4FARBgBENL+VXcWDUdFgMBFGAEQsg4UlWvVviLZbLXjRQAEJsIIgJD17421p/Nm9emq7vFRFlcDoDGEEQAha/7nRyRJNw3jqAgQyAgjAELS/uNl2nzYJYfdppyLUqwuB0ATCCMAQtIHX+RLki7t00VdOkVYXA2AphBGAISkD+q6aMYO7m5xJQDOhTACIOQcPFGujQdLZLOJLhogCBBGAIScBXVdNKMzuqhbrNPiagCcC2EEQMg5dRbN9UPoogGCAWEEQEg5UlKhdXnFkqTrBtNFAwQDwgiAkHKqi2ZUr85Kjou0uBoAzUEYARBSPvi8NoyMpYsGCBqEEQAh46irUqv3F0miiwYIJoQRACFj4eZ8GSMNT09QWgLXogGCBWEEQMiYX9dFc/0QjooAwYQwAiAkFJW5tXLvcUnMugoEG8IIgJDw4baj8hrpwu5xSu8SbXU5APxAGAEQEhZvKZAkZV+YbHElAPxFGAEQ9CqrPfp45zFJ0tcGEUaAYEMYARD0cvccV7nbo+Q4pwanxVldDgA/EUYABL0lW+u6aAYly2azWVwNAH8RRgAENWOMFm85KonxIkCwIowACGqbD7uU76pUdIRDWX26Wl0OgBYgjAAIaovqzqK5on+iIsMdFlcDoCUIIwCC2uIzxosACE6EEQBB63BxhTYfdslmk746MMnqcgC0EGEEQNA6dRbNyJ6d1TXGaXE1AFqKMAIgaC3eylk0QCggjAAISierapS7u/bCeIwXAYIbYQRAUPpkxzG5PV71Tuykvt06WV0OgPNAGAEQlE510VwzMIlZV4EgRxgBEHS8XqOPdtSGka8O4iwaINgRRgAEnc2HXSo86VanCIdG9epidTkAzhNhBEDQWbq99qjI5f0TFRHGnzEg2PEpBhB0ltWFka8MoIsGCAWEEQBB5USZW+sPFEuSvjKgm7XFAGgVhBEAQeXjncdkjDQwJVbd46OsLgdAKyCMAAgqy7Yfk0QXDRBKCCMAgkbtKb2nwghdNECoIIwACBqbDpWoqMytWGeYRvbqbHU5AFoJYQRA0Fi67fQpveEO/nwBoYJPM4Cgsayui+ZqxosAIaVFYWT27NnKyMhQZGSkMjMztWrVqiaXLy4u1pQpU9S9e3c5nU5dcMEFmj9/fosKBtAxHT9ZpU0HiyVJVzFeBAgpYf6uMHfuXE2dOlVz5sxRZmamZs2apZycHG3fvl1JSWf/b8XtdutrX/uakpKS9I9//ENpaWnav3+/EhISWqN+AB3EqVN6L+wep+S4SKvLAdCK/A4jzz33nO666y5NmjRJkjRnzhzNmzdPr776qh5++OGzln/11VdVVFSkFStWKDw8XJKUkZFxflUD6HCWbuMsGiBU+dVN43a7tXbtWmVnZ59+Abtd2dnZys3NbXCdf//738rKytKUKVOUnJyswYMH68knn5TH42n0faqqquRyuerdAHRcHq/RxzvrxosMZLwIEGr8CiOFhYXyeDxKTk6u156cnKz8/PwG19mzZ4/+8Y9/yOPxaP78+Xrsscf0m9/8Rr/85S8bfZ+ZM2cqPj7ed0tPT/enTAAhZtPBYhWXVys2Mkwj0hOsLgdAK2vzs2m8Xq+SkpL0xz/+USNHjtS4ceP0s5/9THPmzGl0nWnTpqmkpMR3O3DgQFuXCSCAfbKzUJJ0eb9EhXFKLxBy/BozkpiYKIfDoYKCgnrtBQUFSklJaXCd7t27Kzw8XA6Hw9c2aNAg5efny+12KyIi4qx1nE6nnE6nP6UBCGGf1HXRXN4/0eJKALQFv/6LERERoZEjR2rJkiW+Nq/XqyVLligrK6vBdS677DLt2rVLXq/X17Zjxw517969wSACAGcqrazW+rxiSdKV/Rm8CoQiv493Tp06VS+//LLeeOMNbd26Vffee6/Kysp8Z9dMmDBB06ZN8y1/7733qqioSPfdd5927NihefPm6cknn9SUKVNabysAhKzP9hSpxmuU0TVa6V2irS4HQBvw+9TecePG6dixY5o+fbry8/M1fPhwLViwwDeoNS8vT3b76YyTnp6uhQsX6oEHHtDQoUOVlpam++67Tw899FDrbQWAkHWqi+YKjooAIctmjDFWF3EuLpdL8fHxKikpUVxcnNXlAGhHVz+7THsLy/THO0bq2osaHpsGIDA19/ubYekAAtaBonLtLSyTw27TpX27Wl0OgDZCGAEQsJbvqj2ld0R6guIiwy2uBkBbIYwACFiMFwE6BsIIgIDk8Rotr5vs7IoLmF8ECGWEEQABadPBYrkqaxQXGaahafFWlwOgDRFGAASkU0dFxvRlCngg1PEJBxCQPqGLBugwCCMAAk5pZbXW5Z2QxBTwQEdAGAEQcJgCHuhYCCMAAs5yTukFOhTCCICAc2q8yOX9GS8CdASEEQAB5UBRufbUTQGfxRTwQIdAGAEQUJgCHuh4CCMAAgpTwAMdD2EEQMDweI0+3XVcEvOLAB0JYQRAwPj8UIlKKqoVyxTwQIdCGAEQMD7ZUdtFcxlTwAMdCp92AAGDKeCBjokwAiAgnKyqYQp4oIMijAAICJ/tPs4U8EAHRRgBEBBOndLLrKtAx0MYARAQfONF6KIBOhzCCADLHTzBFPBAR0YYAWC55TuZAh7oyAgjACzHVXqBjo0wAsBSHq/xXRyP8SJAx0QYAWCpM6eAH9aDKeCBjogwAsBSy3cyBTzQ0fHJB2Cpj5kCHujwCCMALHOyqkbr9tdOAX9FP8aLAB0VYQSAZU5NAd+ra7R6dmUKeKCjIowAsMzps2joogE6MsIIAMt8XDd4lVN6gY6NMALAEgdPlGvPMaaAB0AYAWCRU1PAD2cKeKDDI4wAsMTpq/QyXgTo6AgjANrdmVPAX3kB40WAjo4wAqDdbTpYrJKKasVFhmloGlPAAx0dYQRAuzvVRXNZP6aAB0AYAWCBTzilF8AZCCMA2lVpZbXW5RVLYvAqgFqEEQDtasXu4/J4jfokdlJ6F6aAB0AYAdDOTnfRcFQEQC3CCIB2dXp+EcaLAKhFGAHQbvYfL9P+4+UKdzAFPIDTCCMA2s3HdUdFLu7ZWZ2cYRZXAyBQEEYAtJtPdtSOF2HWVQBnIowAaBfVHq9ydx+XJF3JeBEAZyCMAGgXGw4Uq7SqRp2jw3VRapzV5QAIIIQRAO3iVBfN5f27yW63WVwNgEBCGAHQLj72ndLL/CIA6iOMAGhzxeVubTpYLInxIgDORhgB0OY+3XVcXiNdkByjlPhIq8sBEGAIIwDaHFfpBdAUwgiANmWMOWMKeMaLADgbYQRAm9pTWKZDxRWKCLMrszdTwAM4W4vCyOzZs5WRkaHIyEhlZmZq1apVzVrvrbfeks1m06233tqStwUQhD6uO6V3dEYXRUU4LK4GQCDyO4zMnTtXU6dO1YwZM7Ru3ToNGzZMOTk5Onr0aJPr7du3Tz/5yU90xRVXtLhYAMGHLhoA5+J3GHnuued01113adKkSbrwwgs1Z84cRUdH69VXX210HY/Ho9tvv12PP/64+vTpc14FAwgeVTUe3xTwDF4F0Bi/wojb7dbatWuVnZ19+gXsdmVnZys3N7fR9Z544gklJSVp8uTJzXqfqqoquVyuejcAwWft/hOqqPYoMcapQd1jrS4HQIDyK4wUFhbK4/EoOTm5XntycrLy8/MbXGf58uV65ZVX9PLLLzf7fWbOnKn4+HjfLT093Z8yAQSIZdtrx4tcdUE32WxMAQ+gYW16Nk1paanuuOMOvfzyy0pMbH5/8bRp01RSUuK7HThwoA2rBNBWlm2vHUv2lQF00QBoXJg/CycmJsrhcKigoKBee0FBgVJSUs5afvfu3dq3b59uuukmX5vX661947Awbd++XX379j1rPafTKafT6U9pAALMoeIK7Sg4KbuNKeABNM2vIyMREREaOXKklixZ4mvzer1asmSJsrKyzlp+4MCB+vzzz7Vhwwbf7eabb9bVV1+tDRs20P0ChLBTR0Uu7tlZ8dHhFlcDIJD5dWREkqZOnaqJEydq1KhRGj16tGbNmqWysjJNmjRJkjRhwgSlpaVp5syZioyM1ODBg+utn5CQIElntQMILUu31Y4XuXpgksWVAAh0foeRcePG6dixY5o+fbry8/M1fPhwLViwwDeoNS8vT3Y7E7sCHVlVjUcrdtfOL3LVBXTRAGiazRhjrC7iXFwul+Lj41VSUqK4uDirywFwDst3Fup/XlmppFinVj5yDWfSAB1Uc7+/OYQBoNWdGi/CKb0AmoMwAqDVLa0LI4wXAdAchBEArepAUbl2HyuTw27TZf24Hg2AcyOMAGhVp7poRvbqrPgoTukFcG6EEQCt6tQU8My6CqC5CCMAWk1ltUef1p3Se/UAxosAaB7CCIBWs2pvkSqrvUqJi9TAFK7SC6B5CCMAWs3SMy6Mxym9AJqLMAKgVRhjtHQbV+kF4D/CCIBWsfvYSe07Xq4Ih12Xc5VeAH4gjABoFYu21B4VyerbVTFOvy97BaADI4wAaBWLtxZIkrIvTLa4EgDBhjAC4LwdP1mldXknJEnZgzilF4B/CCMAztuH247KGGlwWpy6x0dZXQ6AIEMYAXDefF00g+iiAeA/wgiA81JZ7dHHO2pnXSWMAGgJwgiA85K7+7gqqj1KiYvURalxVpcDIAgRRgCcl0W+s2iSmHUVQIsQRgC0mNdrtITxIgDOE2EEQIt9cbhEBa4qdYpwKKtvV6vLARCkCCMAWmzxltqjIlde0E3OMIfF1QAIVoQRAC22aGvtFPB00QA4H4QRAC1y8ES5th5xyW6Trh7IrKsAWo4wAqBF/ru5totmZK/O6tIpwuJqAAQzwgiAFvngiyOSpLGDu1tcCYBgRxgB4LcCV6XW7K+9MN51g1MsrgZAsCOMAPDbws35MkYa0TNBqQlcGA/A+SGMAPDb/M9ru2iup4sGQCsgjADwy7HSKq3aWySJLhoArYMwAsAv/92SL6+RhvaIV3qXaKvLARACCCMA/PLB5/mSOIsGQOshjABotqIyt3L3HJckXT+ELhoArYMwAqDZFm3Jl8drdFFqnHp17WR1OQBCBGEEQLPNr+uiuX4IXTQAWg9hBECzlJRX69NdhZKksZxFA6AVEUYANMuirQWq8RoNTIlVn24xVpcDIIQQRgA0ywefcy0aAG2DMALgnIrK3PpoxzFJ0g1D6aIB0LoIIwDOad7nR1TjNRqcFqd+SbFWlwMgxBBGAJzTu+sPSZJuHZ5mcSUAQhFhBECT8o6Xa+3+E7LbpJuHpVpdDoAQRBgB0KR/1R0VuaxfopLiIi2uBkAoIowAaJQxRu9uqA0jXx9BFw2AtkEYAdCojQdLtLewTFHhDuVcxFk0ANoGYQRAo04NXL32omR1coZZXA2AUEUYAdCgao9X7288LEm6lS4aAG2IMAKgQct3Fup4mVtdO0Xoin6JVpcDIIQRRgA06J91XTQ3DUtVmIM/FQDaDn9hAJyltLJa/92cL4mzaAC0PcIIgLN88EW+qmq86tOtk4b2iLe6HAAhjjAC4Cxvrz4gSfrmxT1ks9ksrgZAqCOMAKhn19GTWlM3/fu3RvawuhwAHQBhBEA9b6+pPSry1YFJSmb6dwDtgDACwMdd49U/1x2UJI27pKfF1QDoKAgjAHw+3FagwpNudYt16uoB3awuB0AHQRgB4PPmqtMDV5lbBEB7adFfm9mzZysjI0ORkZHKzMzUqlWrGl325Zdf1hVXXKHOnTurc+fOys7ObnJ5ANbYf7xMH+84JptNum00XTQA2o/fYWTu3LmaOnWqZsyYoXXr1mnYsGHKycnR0aNHG1x+2bJlGj9+vJYuXarc3Fylp6fr2muv1aFDh867eACt528r8yRJV13QTT27RltcDYCOxGaMMf6skJmZqUsuuUQvvfSSJMnr9So9PV0/+tGP9PDDD59zfY/Ho86dO+ull17ShAkTmvWeLpdL8fHxKikpUVxcnD/lAmiGymqPLp25RMXl1Xpl4ihdMyjZ6pIAhIDmfn/7dWTE7XZr7dq1ys7OPv0Cdruys7OVm5vbrNcoLy9XdXW1unTp0ugyVVVVcrlc9W4A2s68TUdUXF6ttIQofWVAktXlAOhg/AojhYWF8ng8Sk6u/7+m5ORk5efnN+s1HnroIaWmptYLNF82c+ZMxcfH+27p6en+lAnAT3/5bL8k6bbMnnLYmXEVQPtq1+HyTz31lN566y3961//UmRk45MpTZs2TSUlJb7bgQMH2rFKoGNZl3dCGw4UK8Jh17hLCP4A2l+YPwsnJibK4XCooKCgXntBQYFSUlKaXPfZZ5/VU089pcWLF2vo0KFNLut0OuV0Ov0pDUALvfbpPknSzcNTlRjD5w5A+/PryEhERIRGjhypJUuW+Nq8Xq+WLFmirKysRtd75pln9Itf/EILFizQqFGjWl4tgFZ1uLhC8z8/Ikn6/mW9La4GQEfl15ERSZo6daomTpyoUaNGafTo0Zo1a5bKyso0adIkSdKECROUlpammTNnSpKefvppTZ8+XW+++aYyMjJ8Y0tiYmIUExPTipsCwF9/zt0vj9coq09XXZjKmWoArOF3GBk3bpyOHTum6dOnKz8/X8OHD9eCBQt8g1rz8vJkt58+4PL73/9ebrdb3/rWt+q9zowZM/Tzn//8/KoH0GLl7hr9fVXt3CLfv5yjIgCs4/c8I1ZgnhGg9b326V49/v4W9eoarQ8f/Apn0QBodW0yzwiA0FDt8epPn+yVJN11RR+CCABLEUaADuj9jYd1qLhCiTFOfWtkD6vLAdDBEUaADsbrNZrz0W5J0qTLMhQZ7rC4IgAdHWEE6GCWbj+qHQUnFeMM0/9c2svqcgCAMAJ0JMYY/XbJTknS7Zf2VHxUuMUVAQBhBOhQlm0/po0HSxQV7tBdV/SxuhwAkEQYAToMY4xmLd4hSZqQ1Yup3wEEDMII0EHUOypyJUdFAAQOwgjQAXi9Rs8tqj0qcgdHRQAEGMII0AHM/+KIPj9Uok4RDt3NUREAAYYwAoS4ao9Xzy7cLkm668o+HBUBEHAII0CIm7v6gPYdL1fXThG6kzNoAAQgwggQwsqqavRC3bwiP/pqP8U4/b5QNwC0OcIIEMJ+t2yXjpVWqVfXaN2WyWyrAAITYQQIUQeKyvVy3ZV5f3b9IEWE8XEHEJj46wSEqJkfbJW7xqvL+nXV1y5MtrocAGgUYQQIQSt2FWr+5/my26THbrxQNpvN6pIAoFGEESDEVNV49Oi7X0iS/ufSXhqYEmdxRQDQNMIIEGLmLNujPYVl6hbr1E9yBlhdDgCcE2EECCF7C8s0e9kuSdL0Gy9UXGS4xRUBwLkRRoAQ4fUaPfT/Nsld49UV/RN149DuVpcEAM1CGAFCxBu5+7Rqb5GiIxx68utDGLQKIGgQRoAQsK+wTE8v2CZJmjZ2oNK7RFtcEQA0H2EECHI1Hq9+8s5GVVZ7ldWnq25nplUAQYYwAgS5Fz/cpTX7TyjGGaZnvjVUdjvdMwCCC2EECGKr9xXpxQ9rL4T3q68PpnsGQFAijABB6kSZW/e/tUFeI31jRJpuGZ5mdUkA0CKEESAIebxGP35rvQ4VVyija7SeuHWw1SUBQIsRRoAgNGvxDn2ys1CR4XbNuWOkYpxhVpcEAC1GGAGCzIIvjujFD2tnWX3qG0O59gyAoEcYAYLIpoPFun/uBknS98Zk6NYRjBMBEPwII0CQOFJSoTvfWKPKaq++MqCbHr1hkNUlAUCrIIwAQeBEmVsTXlmlo6VVGpAcqxfHj1CYg48vgNDAXzMgwJVV1WjS66u18+hJpcRF6tVJlyiWq/ECCCGEESCAVbg9uvsva7ThQLESosP1l8mjlZYQZXVZANCqCCNAgKpwe3Tnn1fr013HFR3h0Gvfu0T9k2OtLgsAWh2TEwABqKyqRnf9eY1W7D6uThEOvfH90RrRs7PVZQFAmyCMAAGmqMytSa+t0saDJb4gMiqji9VlAUCbIYwAAeRAUbkmvrZKe46VqXN0uF6bNFrD0xOsLgsA2hRhBAgQq/cV6Qd/WauiMrdS4yP158mZ6pcUY3VZANDmCCOAxYwxmrv6gB577wtVe4wGp8XpTxMuUUp8pNWlAUC7IIwAFqpwe/TYe1/oH2sPSpLGDk7Rc98ZrqgIh8WVAUD7IYwAFtly2KX7567XjoKTstukB68doHuv6iu73WZ1aQDQrggjQDvzeI3+9Mke/ea/O+T2eJUYE6Hfjh+hMX0TrS4NACxBGAHa0dYjLj38/zZp48ESSVL2oGQ99c0hSoxxWlwZAFiHMAK0gxNlbr2wZKf++tl+1XiNYiPD9LPrB2ncJemy2eiWAdCxEUaANlRV49EbK/bpxQ93qbSyRpJ03UUpevyWi5Qcx9kyACARRoA2UePxat7nR/Tsf7frQFGFJGlgSqweveFCXd6fsSEAcCbCCNCKKqs9emfNAb38yV7lFZVLkpJinfpJzgB98+IecnCmDACchTACtIJjpVV6a1WeXl+xT8fL3JKkLp0iNGlMhiZf0VvREXzUAKAx/IUEWqjG49XS7cf09poD+nDbUXm8RpLUo3OU7r6yj749Mp3JywCgGQgjgB+8XqN1eSe04It8vbfxsI6VVvmeG9EzQd8bk6EbhnRXmMNuYZUAEFwII8A5VFZ7tHJvkRZuzteiLQX1AkhiTIS+cXEPfXtkD/VPjrWwSgAIXoQR4Es8XqPPD5Xo012FWrG7UGv2nVBVjdf3fKwzTNcMStLYId311YFJCucoCACcF8IIOrxjpVXacKBY6/NOaMOBYm08UKwyt6feMkmxTl0zKFnXDU5RVp+uiggjgABAayGMoMMoqajWzoJS7Sg4qR0FpXW3kyo8WXXWsrGRYcrq01WX9UvUZf26qm+3GGZKBYA20qIwMnv2bP36179Wfn6+hg0bphdffFGjR49udPl33nlHjz32mPbt26f+/fvr6aef1vXXX9/iooGGVHu8yi+p1IGich08UaGDJ8p1oO5nXlG5Clxnhw5Jstmk/kkxGpHeWcN7JmhEzwT1T4plThAAaCd+h5G5c+dq6tSpmjNnjjIzMzVr1izl5ORo+/btSkpKOmv5FStWaPz48Zo5c6ZuvPFGvfnmm7r11lu1bt06DR48uFU2AqGpstojV0W1Sr50Kypz61hpVe3tZJXvflG5W8Y0/Zrd4yPVPzlWA5Jj6n7Gql9SjDo5OUgIAFaxGXOuP9/1ZWZm6pJLLtFLL70kSfJ6vUpPT9ePfvQjPfzww2ctP27cOJWVlek///mPr+3SSy/V8OHDNWfOnGa9p8vlUnx8vEpKShQXF+dPuWhlxhi5PV5Ve4yqa7xye7xy1/2s9nhVXWPk9njkrjG+5yqqPapw16jc7VG526OKUz+r67dVVHtUWnk6dFRWe89d0JdEhNnVo3OUenSOVo/OUUqv+9mjc5T6dItRfFR4G/xWAAANae73t1//HXS73Vq7dq2mTZvma7Pb7crOzlZubm6D6+Tm5mrq1Kn12nJycvTuu+82+j5VVVWqqjp9SN3lcvlTZrP96ZM9Onii9rohxhidSmXGSKce1d4/3a4z2+ueMDJn3D/7NVTvNb78Pme368vv38j7mDNWOLPd4zXyGiOP19S/b2rnyajfZmrbjJHXq7PaPN7a1/Z4jWq8tSGkPdlsUlxkuOKjam8J0eFKiI5QUqxT3WKd6hZT+zMpzqnEGKe6REfITvcKAAQVv8JIYWGhPB6PkpOT67UnJydr27ZtDa6Tn5/f4PL5+fmNvs/MmTP1+OOP+1Nai8z7/IjW5xW3+fuEMofdpnCHTREOuyLC7Ipw2BUeZle44/T9qHC7oiPCFBXhUHS4Q9ERDkVFhCk6ovZ+ZF1bdIRDMc7ToSMuKlyxzjDCBQCEuIDsKJ82bVq9oykul0vp6emt/j7fGtlDY/p2lU21X3Y2m+T72rPZfPdr2+svc+rEijPPsGhouYbaTy//5fc4o/3U65+xUr33la3R97DbbXLYJbvNJofdJofNVttW99jua9NZbQ677fR69tpaTrWH1YWO8FOhw2FnkCcA4Lz5FUYSExPlcDhUUFBQr72goEApKSkNrpOSkuLX8pLkdDrldDr9Ka1Fbs/s1ebvAQAAmubXzE0REREaOXKklixZ4mvzer1asmSJsrKyGlwnKyur3vKStGjRokaXBwAAHYvf3TRTp07VxIkTNWrUKI0ePVqzZs1SWVmZJk2aJEmaMGGC0tLSNHPmTEnSfffdp6uuukq/+c1vdMMNN+itt97SmjVr9Mc//rF1twQAAAQlv8PIuHHjdOzYMU2fPl35+fkaPny4FixY4BukmpeXJ7v99AGXMWPG6M0339Sjjz6qRx55RP3799e7777LHCMAAEBSC+YZsQLzjAAAEHya+/3N1b4AAIClCCMAAMBShBEAAGApwggAALAUYQQAAFiKMAIAACxFGAEAAJYijAAAAEsRRgAAgKX8ng7eCqcmiXW5XBZXAgAAmuvU9/a5JnsPijBSWloqSUpPT7e4EgAA4K/S0lLFx8c3+nxQXJvG6/Xq8OHDio2Nlc1ma7XXdblcSk9P14EDB0L2mjdsY/AL9e2T2MZQEOrbJ4X+NrbF9hljVFpaqtTU1HoX0f2yoDgyYrfb1aNHjzZ7/bi4uJD8h3UmtjH4hfr2SWxjKAj17ZNCfxtbe/uaOiJyCgNYAQCApQgjAADAUh06jDidTs2YMUNOp9PqUtoM2xj8Qn37JLYxFIT69kmhv41Wbl9QDGAFAAChq0MfGQEAANYjjAAAAEsRRgAAgKUIIwAAwFIhH0Z+9atfacyYMYqOjlZCQkKDy+Tl5emGG25QdHS0kpKS9NOf/lQ1NTVNvm5RUZFuv/12xcXFKSEhQZMnT9bJkyfbYAv8s2zZMtlstgZvq1evbnS9r3zlK2ctf88997Rj5c2XkZFxVq1PPfVUk+tUVlZqypQp6tq1q2JiYvTNb35TBQUF7VSxf/bt26fJkyerd+/eioqKUt++fTVjxgy53e4m1wv0fTh79mxlZGQoMjJSmZmZWrVqVZPLv/POOxo4cKAiIyM1ZMgQzZ8/v50q9d/MmTN1ySWXKDY2VklJSbr11lu1ffv2Jtd5/fXXz9pfkZGR7VSxf37+85+fVevAgQObXCeY9p/U8N8Vm82mKVOmNLh8MOy/jz/+WDfddJNSU1Nls9n07rvv1nveGKPp06ere/fuioqKUnZ2tnbu3HnO1/X3s9wcIR9G3G63vv3tb+vee+9t8HmPx6MbbrhBbrdbK1as0BtvvKHXX39d06dPb/J1b7/9dm3evFmLFi3Sf/7zH3388ce6++6722IT/DJmzBgdOXKk3u3OO+9U7969NWrUqCbXveuuu+qt98wzz7RT1f574okn6tX6ox/9qMnlH3jgAb3//vt655139NFHH+nw4cP6xje+0U7V+mfbtm3yer36wx/+oM2bN+v555/XnDlz9Mgjj5xz3UDdh3PnztXUqVM1Y8YMrVu3TsOGDVNOTo6OHj3a4PIrVqzQ+PHjNXnyZK1fv1633nqrbr31Vn3xxRftXHnzfPTRR5oyZYo+++wzLVq0SNXV1br22mtVVlbW5HpxcXH19tf+/fvbqWL/XXTRRfVqXb58eaPLBtv+k6TVq1fX275FixZJkr797W83uk6g77+ysjINGzZMs2fPbvD5Z555Rr/97W81Z84crVy5Up06dVJOTo4qKysbfU1/P8vNZjqI1157zcTHx5/VPn/+fGO3201+fr6v7fe//72Ji4szVVVVDb7Wli1bjCSzevVqX9sHH3xgbDabOXToUKvXfj7cbrfp1q2beeKJJ5pc7qqrrjL33Xdf+xR1nnr16mWef/75Zi9fXFxswsPDzTvvvONr27p1q5FkcnNz26DC1vfMM8+Y3r17N7lMIO/D0aNHmylTpvgeezwek5qaambOnNng8t/5znfMDTfcUK8tMzPT/OAHP2jTOlvL0aNHjSTz0UcfNbpMY3+TAtGMGTPMsGHDmr18sO8/Y4y57777TN++fY3X623w+WDaf8YYI8n861//8j32er0mJSXF/PrXv/a1FRcXG6fTaf7+9783+jr+fpabK+SPjJxLbm6uhgwZouTkZF9bTk6OXC6XNm/e3Og6CQkJ9Y40ZGdny263a+XKlW1esz/+/e9/6/jx45o0adI5l/3b3/6mxMREDR48WNOmTVN5eXk7VNgyTz31lLp27aoRI0bo17/+dZPdamvXrlV1dbWys7N9bQMHDlTPnj2Vm5vbHuWet5KSEnXp0uWcywXiPnS73Vq7dm2937/dbld2dnajv//c3Nx6y0u1n8tg2l+SzrnPTp48qV69eik9PV233HJLo39zAsHOnTuVmpqqPn366Pbbb1deXl6jywb7/nO73frrX/+q73//+01enDWY9t+X7d27V/n5+fX2U3x8vDIzMxvdTy35LDdXUFwory3l5+fXCyKSfI/z8/MbXScpKaleW1hYmLp06dLoOlZ55ZVXlJOTc84LDd52223q1auXUlNTtWnTJj300EPavn27/vnPf7ZTpc334x//WBdffLG6dOmiFStWaNq0aTpy5Iiee+65BpfPz89XRETEWWOGkpOTA25/NWTXrl168cUX9eyzzza5XKDuw8LCQnk8ngY/Z9u2bWtwncY+l8Gwv7xer+6//35ddtllGjx4cKPLDRgwQK+++qqGDh2qkpISPfvssxozZow2b97cphcGbYnMzEy9/vrrGjBggI4cOaLHH39cV1xxhb744gvFxsaetXww7z9Jevfdd1VcXKzvfe97jS4TTPuvIaf2hT/7qSWf5eYKyjDy8MMP6+mnn25yma1bt55zgFUwack2Hzx4UAsXLtTbb799ztc/c7zLkCFD1L17d11zzTXavXu3+vbt2/LCm8mf7Zs6daqvbejQoYqIiNAPfvADzZw5M6CnaW7JPjx06JCuu+46ffvb39Zdd93V5LpW70PUmjJlir744osmx1RIUlZWlrKysnyPx4wZo0GDBukPf/iDfvGLX7R1mX4ZO3as7/7QoUOVmZmpXr166e2339bkyZMtrKxtvPLKKxo7dqxSU1MbXSaY9l8wCMow8uCDDzaZWCWpT58+zXqtlJSUs0YCnzrLIiUlpdF1vjxYp6amRkVFRY2uc75ass2vvfaaunbtqptvvtnv98vMzJRU+7/y9vgiO599mpmZqZqaGu3bt08DBgw46/mUlBS53W4VFxfXOzpSUFDQZvurIf5u4+HDh3X11VdrzJgx+uMf/+j3+7X3PmxMYmKiHA7HWWcvNfX7T0lJ8Wv5QPHDH/7QN6Dd3/8dh4eHa8SIEdq1a1cbVdd6EhISdMEFFzRaa7DuP0nav3+/Fi9e7PcRxWDaf9Lp77eCggJ1797d115QUKDhw4c3uE5LPsvNdl4jToLIuQawFhQU+Nr+8Ic/mLi4OFNZWdnga50awLpmzRpf28KFCwNqAKvX6zW9e/c2Dz74YIvWX758uZFkNm7c2MqVtb6//vWvxm63m6KiogafPzWA9R//+Ievbdu2bQE9gPXgwYOmf//+5rvf/a6pqalp0WsE0j4cPXq0+eEPf+h77PF4TFpaWpMDWG+88cZ6bVlZWQE7ANLr9ZopU6aY1NRUs2PHjha9Rk1NjRkwYIB54IEHWrm61ldaWmo6d+5sXnjhhQafD7b9d6YZM2aYlJQUU11d7dd6gb7/1MgA1meffdbXVlJS0qwBrP58lptd33mtHQT2799v1q9fbx5//HETExNj1q9fb9avX29KS0uNMbX/gAYPHmyuvfZas2HDBrNgwQLTrVs3M23aNN9rrFy50gwYMMAcPHjQ13bdddeZESNGmJUrV5rly5eb/v37m/Hjx7f79jVm8eLFRpLZunXrWc8dPHjQDBgwwKxcudIYY8yuXbvME088YdasWWP27t1r3nvvPdOnTx9z5ZVXtnfZ57RixQrz/PPPmw0bNpjdu3ebv/71r6Zbt25mwoQJvmW+vH3GGHPPPfeYnj17mg8//NCsWbPGZGVlmaysLCs24ZwOHjxo+vXrZ6655hpz8OBBc+TIEd/tzGWCaR++9dZbxul0mtdff91s2bLF3H333SYhIcF3Ftsdd9xhHn74Yd/yn376qQkLCzPPPvus2bp1q5kxY4YJDw83n3/+uVWb0KR7773XxMfHm2XLltXbX+Xl5b5lvryNjz/+uFm4cKHZvXu3Wbt2rfnud79rIiMjzebNm63YhCY9+OCDZtmyZWbv3r3m008/NdnZ2SYxMdEcPXrUGBP8++8Uj8djevbsaR566KGzngvG/VdaWur7zpNknnvuObN+/Xqzf/9+Y4wxTz31lElISDDvvfee2bRpk7nllltM7969TUVFhe81vvrVr5oXX3zR9/hcn+WWCvkwMnHiRCPprNvSpUt9y+zbt8+MHTvWREVFmcTERPPggw/WS8VLly41kszevXt9bcePHzfjx483MTExJi4uzkyaNMkXcALB+PHjzZgxYxp8bu/evfV+B3l5eebKK680Xbp0MU6n0/Tr18/89Kc/NSUlJe1YcfOsXbvWZGZmmvj4eBMZGWkGDRpknnzyyXpHsb68fcYYU1FRYf73f//XdO7c2URHR5uvf/3r9b7cA8lrr73W4L/ZMw9kBuM+fPHFF03Pnj1NRESEGT16tPnss898z1111VVm4sSJ9ZZ/++23zQUXXGAiIiLMRRddZObNm9fOFTdfY/vrtdde8y3z5W28//77fb+P5ORkc/3115t169a1f/HNMG7cONO9e3cTERFh0tLSzLhx48yuXbt8zwf7/jtl4cKFRpLZvn37Wc8F4/479d315dup7fB6veaxxx4zycnJxul0mmuuueasbe/Vq5eZMWNGvbamPsstZTPGmPPr6AEAAGi5Dj/PCAAAsBZhBAAAWIowAgAALEUYAQAAliKMAAAASxFGAACApQgjAADAUoQRAABgKcIIAACwFGEEAABYijACAAAsRRgBAACW+v84bpZZqD/2QAAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create an array of numbers from -10 to 10\n",
    "nn_outputs = np.arange(-10,10,.1)\n",
    "# Define the sigmoid function\n",
    "sigmoid = lambda x: 1 / (1 + np.exp(-x))\n",
    "\n",
    "plt.plot(nn_outputs, sigmoid(nn_outputs))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Our network will output continuous real numbers in the last layer, and the sigmoid function will transform them into probabilities from `0` to `1`.  The sigmoid will output the probability that the example belongs to a given class.  For example, if we code a star as `1`, and everything else as a `0`, the output of the sigmoid activation tells us the probability that an example is a star.  The sigmoid output might be `.51`, corresponding to a 51% probability.\n",
    "\n",
    "## Negative Log Likelihood\n",
    "\n",
    "Because the range of values after we apply the sigmoid function is so small, mean squared error isn't the best loss function.  If the actual target is a `1`, and we predict a `.5`, mean squared error will only be `.25`.  Here is how mean squared error looks with binary classification:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "outputs": [
    {
     "data": {
      "text/plain": "[<matplotlib.lines.Line2D at 0x1775fad70>]"
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+aElEQVR4nO3deVxVdeLG8efeC1xcABeURVHczQ0VlXBpG8qybKwpnSw1szKzpmSmkkqdptJWa0rT0TRtdansV+lYRjlumIlSmrii4gaIC6ts957fHxZFaXIRONzL5/16nRd6PIf73BPDfeac7/kei2EYhgAAAExiNTsAAACo3SgjAADAVJQRAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTeZkdoDycTqeOHj0qPz8/WSwWs+MAAIByMAxDOTk5Cg0NldV6/vMfblFGjh49qrCwMLNjAACACjh06JCaN29+3n93izLi5+cn6eyb8ff3NzkNAAAoj+zsbIWFhZV+jp+PW5SRny/N+Pv7U0YAAHAzFxpiwQBWAABgKsoIAAAwFWUEAACYijICAABMRRkBAACmoowAAABTUUYAAICpKCMAAMBUlBEAAGAql8vImjVrNHjwYIWGhspiseiTTz654D6rV69Wz549Zbfb1bZtWy1YsKACUQEAgCdyuYzk5eUpIiJCM2fOLNf2+/fv1/XXX68rr7xSSUlJevjhh3X33Xfriy++cDksAADwPC4/m+a6667TddddV+7tZ8+erVatWunll1+WJF1yySVat26dXnnlFQ0cONDVlwcAAB6myseMJCQkKCYmpsy6gQMHKiEh4bz7FBYWKjs7u8xSFdbtydTI+ZtUUOyoku8PAAAurMrLSFpamoKCgsqsCwoKUnZ2ts6cOXPOfaZNm6aAgIDSJSwsrNJzFRQ7NGFJktbsPq5XVu2u9O8PAADKp0beTRMXF6esrKzS5dChQ5X+Gr7eNk29qaskac7aFG0+cLLSXwMAAFxYlZeR4OBgpaenl1mXnp4uf39/1alT55z72O12+fv7l1mqwtWdgnRLZHMZhvT3pd8rv6ikSl4HAACcX5WXkejoaMXHx5dZt2rVKkVHR1f1S5fL5MGdFBrgq4Mn8vXcf3eaHQcAgFrH5TKSm5urpKQkJSUlSTp7625SUpJSU1Mlnb3EMnLkyNLt77vvPqWkpOjRRx/Vzp079cYbb2jJkiWaMGFC5byDi+Tv660XbomQJL2dcFDr92aanAgAgNrF5TKyefNm9ejRQz169JAkxcbGqkePHpo8ebIk6dixY6XFRJJatWql5cuXa9WqVYqIiNDLL7+sN998s0bd1tu/XaBGXNpSkvTohz8ou6DY5EQAANQeFsMwDLNDXEh2drYCAgKUlZVVZeNH8gpLdN2/1yr1ZL6G9mpeerYEAABUTHk/v2vk3TRmqGf30ku3RshikZZsPqz45PQL7wQAAC4aZeRX+rRqpLv7t5IkTfx4m07lFZmcCAAAz0cZ+Y2/X9NBbZvW1/GcQk3+9Eez4wAA4PEoI7/h623Ty7dGyGa16LPvj+rzH46aHQkAAI9GGTmHiLAGGn9FG0nSpE+2KyOnwOREAAB4LsrIeTxwVTt1CvHXqfxiPf7xdrnBTUcAALglysh5+HhZNX1YhLxtFn2VnK6PthwxOxIAAB6JMvIHOgb7a8LV7SVJT332o46ePvdThgEAQMVRRi7g3gGt1T2sgXIKSvTYRz9wuQYAgEpGGbkAL5tVLw+NkN3LqrV7MvXOxoNmRwIAwKNQRsqhTZP6iruuoyRp6opk7c3INTkRAACegzJSTiOjwzWgXaAKip2KXZKkYofT7EgAAHgEykg5Wa0WvXhLhALqeOuHw1l6/eu9ZkcCAMAjUEZcEBzgq2dv6iJJmvnNXm1JPWVyIgAA3B9lxEU3dAvVkO6hcjgNxS5OUl5hidmRAABwa5SRCnjqz10UEuCrAyfy9eyKZLPjAADg1igjFRBQx1sv3xohSXr/21R9vTPd5EQAALgvykgF9W0bqDH9W0mSHv1wm07kFpqcCAAA90QZuQiPDOygdk3rKzO3UHEfb2N2VgAAKoAychF8vW169a/d5W2z6Msd6fow8bDZkQAAcDuUkYvUOTRAsVd3kCQ99dkOHTqZb3IiAADcC2WkEtx7WWv1Dm+o3MIS/X3J93I4uVwDAEB5UUYqgc1q0fSh3VXPx6ZNB05qzpoUsyMBAOA2KCOVJKxRXU25sbMkafqqXfrxaJbJiQAAcA+UkUp0a2RzXdMpSMUOQw8vSlJBscPsSAAA1HiUkUpksVg07eauauJn156MXE1ldlYAAC6IMlLJGte3l87O+nbCQcUnMzsrAAB/hDJSBS5r3+RXs7P+oIycApMTAQBQc1FGqsgjAzuoY7CfTuQV6ZGlPzA7KwAA50EZqSK+3ja9dlsP2b2s+t/u41qw4YDZkQAAqJEoI1WofZCfnrj+EknStP/u1M60bJMTAQBQ81BGqtiIS1vqqo5NVVTi1EMfcLsvAAC/RRmpYhaLRS/c0k2B9X20Kz1Hz/13p9mRAACoUSgj1SCwvl0v/nS774INB/TNrgyTEwEAUHNQRqrJlR2a6s6+4ZKkR5b+oMzcQnMDAQBQQ1BGqtHE6zqqQ5CfMnML9eiH3O4LAIBEGalWvt42/fu27vLxsurrnRl6Z+NBsyMBAGA6ykg16xjsr7jrOkqSnl2erN3pOSYnAgDAXJQRE9zZN1xXdGiiwhKnHnx/K7f7AgBqNcqICSwWi168JUKB9e3alZ6jZ5bvMDsSAACmoYyYpImfXdOHnr3d992NqVq5/ZjJiQAAMAdlxESXtW+isZe3lnT26b5HTp8xOREAANWPMmKyf1zTQRFhDZRdUKKHF21VicNpdiQAAKoVZcRk3jarXv9rD9W3e+m7A6f02td7zY4EAEC1oozUAC0a19WzN3WRJM34eo82ppwwOREAANWHMlJD/Ll7M90a2VxOQ3p4UZJO5RWZHQkAgGpBGalB/nljZ7VuUk9p2QV6hOniAQC1BGWkBqln99Lrt/WQj82qr5LT9XYC08UDADwfZaSG6RwaoLhBP00XvyJZO45mm5wIAICqRRmpge7sG64/dWyqohKnHvxgi/KLSsyOBABAlaGM1EAWi0Uv3hqhIH+79h3P01OfMl08AMBzUUZqqEb1fPTKsO6yWKTFmw/p0++Pmh0JAIAqQRmpwfq2CdQDV7aVJMV99IP2Z+aZnAgAgMpHGanhHvpTO/Vp1Uh5RQ7d/94WFRQ7zI4EAECloozUcF42q16/rYca1/NR8rFsPf0540cAAJ6FMuIGgvx9S8ePvPdtKuNHAAAepUJlZObMmQoPD5evr6+ioqK0adOmP9z+1VdfVYcOHVSnTh2FhYVpwoQJKigoqFDg2uqy9k00/grGjwAAPI/LZWTx4sWKjY3VlClTtGXLFkVERGjgwIHKyMg45/bvv/++Jk6cqClTpig5OVnz5s3T4sWL9fjjj190+Nrm4RjGjwAAPI/LZWT69Om65557NHr0aHXq1EmzZ89W3bp1NX/+/HNuv2HDBvXr10/Dhw9XeHi4rrnmGt12220XPJuC32P8CADAE7lURoqKipSYmKiYmJhfvoHVqpiYGCUkJJxzn759+yoxMbG0fKSkpGjFihUaNGjQRcSuvRg/AgDwNF6ubJyZmSmHw6GgoKAy64OCgrRz585z7jN8+HBlZmaqf//+MgxDJSUluu+++/7wMk1hYaEKCwtL/56dzfNZfu3n8SMzvtmruI9+UNdmAWoVWM/sWAAAVEiV302zevVqTZ06VW+88Ya2bNmijz/+WMuXL9fTTz993n2mTZumgICA0iUsLKyqY7odxo8AADyFS2UkMDBQNptN6enpZdanp6crODj4nPtMmjRJI0aM0N13362uXbvqpptu0tSpUzVt2jQ5nc5z7hMXF6esrKzS5dChQ67ErBUYPwIA8BQulREfHx9FRkYqPj6+dJ3T6VR8fLyio6PPuU9+fr6s1rIvY7PZJEmGYZxzH7vdLn9//zILfo/xIwAAT+DyZZrY2FjNnTtXCxcuVHJyssaNG6e8vDyNHj1akjRy5EjFxcWVbj948GDNmjVLixYt0v79+7Vq1SpNmjRJgwcPLi0lqDjmHwEAuDuXBrBK0rBhw3T8+HFNnjxZaWlp6t69u1auXFk6qDU1NbXMmZAnn3xSFotFTz75pI4cOaImTZpo8ODBevbZZyvvXdRyD8e006YDJ7Vp/0nd/94WLbu/r3y9KXoAAPdgMc53raQGyc7OVkBAgLKysrhkcx7p2QUa9O+1OpFXpNujWujZm7qaHQkAUMuV9/ObZ9N4CMaPAADcFWXEg/x2/MjejFyTEwEAcGGUEQ/zcEw7RZXOP5Ko/KISsyMBAPCHKCMexstm1evDe6iJn12703MV9/G2895CDQBATUAZ8UBN/Xw1c3hP2awW/V/SUb278aDZkQAAOC/KiIfq06qRJl7bUZL0r893aGvqKZMTAQBwbpQRD3b3gFa6tnOwih2Gxr+3RSfzisyOBADA71BGPJjFYtGLt3ZTq8B6OppVoIcWbZXDyfgRAEDNQhnxcH6+3pp1R0/5elu1dk+mXovfY3YkAADKoIzUAh2D/TX1pxlZX/t6j1bvyjA5EQAAv6CM1BI392yu26NayDCkhxcn6fCpfLMjAQAgiTJSq0we3EndmgfodH6x7n9viwpLHGZHAgCAMlKb2L1smjm8pxrU9dYPh7P09Oc7zI4EAABlpLYJa1RXr/70QL13N6Zq2dbDZkcCANRylJFa6IoOTfXgVe0kSXEfb1PysWyTEwEAajPKSC310J/aaUC7QBUUOzX2nUSdzmdCNACAOSgjtZTNatFrf+2h5g3rKPVkvh5alMSEaAAAU1BGarGG9Xz0nxGR8vW26n+7j+uVVbvNjgQAqIUoI7Vc59AAPXdzN0nSjG/2auX2NJMTAQBqG8oINKRHM93Vr5Uk6e9LkrQ3I8fkRACA2oQyAklS3KCOimrVSHlFDt37TqJyCorNjgQAqCUoI5Akedusmnl7T4UE+CrleJ5il3wvJwNaAQDVgDKCUoH17Zp9R6R8vKxatSNdM7/Za3YkAEAtQBlBGRFhDfTMn7tIkqZ/tVvf7OQJvwCAqkUZwe8M7R1W+oTfvy3aqgOZeWZHAgB4MMoIzmnK4M7q2aKBcgpKNPadROUVlpgdCQDgoSgjOCcfL6tm3RGpJn527UrP0aMf/SDDYEArAKDyUUZwXkH+vpp1e095WS1a/sMxzVmTYnYkAIAHoozgD/UKb6QpgztJkp5fuVPr9mSanAgA4GkoI7igOy5tqVsim8tpSA98sEWpJ/LNjgQA8CCUEVyQxWLRM0O6qFvzAJ3OL9Y9b29WLgNaAQCVhDKCcvH1tmnOiF6lA1onLE5ihlYAQKWgjKDcggN8NWfELzO0Tl+12+xIAAAPQBmBS3q0aKjnbu4qSZrxzV599v1RkxMBANwdZQQuu7lnc429rLUk6ZEPv9e2w1kmJwIAuDPKCCrk0Ws76ooOTVRQ7NS972xWRk6B2ZEAAG6KMoIKsVkteu22HmrdpJ6OZRVo7DuJKixxmB0LAOCGKCOoMH9fb80b1Vv+vl7amnpaTyzbzpTxAACXUUZwUVoF1tPM23vKapE+TDyseev2mx0JAOBmKCO4aAPaNdGT15+dMn7qimT9b/dxkxMBANwJZQSVYnS/cA3t9dOU8e9v0b7juWZHAgC4CcoIKoXFYtHTQ7oosmVD5RSU6J6Fm5V1ptjsWAAAN0AZQaWxe9k0+45IhQb4KiUzTw9+sFUOpowHAFwAZQSVqomfXXNG9pKvt1Vrdh/XtBXJZkcCANRwlBFUui7NAvTyrd0lSW+u26+lmw+ZGwgAUKNRRlAlru8Wor9d1VaS9MSy7Uo8eNLkRACAmooygirzcEx7DewcpCKHU/e+nahDJ/PNjgQAqIEoI6gyVqtF04d2V6cQf53IK9KYhd8pp4A7bAAAZVFGUKXq2b00785eaupn1+70XD34wVaVOJxmxwIA1CCUEVS5kIA6enPU2TtsVu86rmeWc4cNAOAXlBFUi27NG+iVod0lSQs2HNDbCQdMzQMAqDkoI6g213UN0SMDO0iSnvpsB8+wAQBIooygmt1/RRv9pWdzOZyGHnhvi3an55gdCQBgMsoIqpXFYtHUm7uoT3gj5RSW6K4F3+lEbqHZsQAAJqKMoNrZvWyaPSJSLRvX1eFTZ3TvO4kqKHaYHQsAYBLKCEzRqJ6P5o3qLX9fLyUePKV/LP1eTh6qBwC1EmUEpmnbtL5mj4iUt82iz384ppe+3GV2JACACSpURmbOnKnw8HD5+voqKipKmzZt+sPtT58+rfHjxyskJER2u13t27fXihUrKhQYnqVvm0BNu7mbJOmN1fu0aFOqyYkAANXN5TKyePFixcbGasqUKdqyZYsiIiI0cOBAZWRknHP7oqIiXX311Tpw4IA+/PBD7dq1S3PnzlWzZs0uOjw8wy2RzfW3P7WTJD3xyXat3cMtvwBQm1gMw3DpQn1UVJR69+6tGTNmSJKcTqfCwsL04IMPauLEib/bfvbs2XrxxRe1c+dOeXt7Vyhkdna2AgIClJWVJX9//wp9D9RshmEodsn3Wrb1iPzsXvpwXF91CPYzOxYA4CKU9/PbpTMjRUVFSkxMVExMzC/fwGpVTEyMEhISzrnPp59+qujoaI0fP15BQUHq0qWLpk6dKofj/HdPFBYWKjs7u8wCz2axWPTcX7qqT6uzt/yOfmuTMrILzI4FAKgGLpWRzMxMORwOBQUFlVkfFBSktLS0c+6TkpKiDz/8UA6HQytWrNCkSZP08ssv65lnnjnv60ybNk0BAQGlS1hYmCsx4absXjbNGRGp1k3q6WhWgcYs3Kz8ohKzYwEAqliV303jdDrVtGlTzZkzR5GRkRo2bJieeOIJzZ49+7z7xMXFKSsrq3Q5dOhQVcdEDdGgro8W3NlHjev5aNuRLP3tg61ycMsvAHg0l8pIYGCgbDab0tPTy6xPT09XcHDwOfcJCQlR+/btZbPZStddcsklSktLU1FR0Tn3sdvt8vf3L7Og9mjRuK7mjuolu5dVXyVn6OnPd8jFoU0AADfiUhnx8fFRZGSk4uPjS9c5nU7Fx8crOjr6nPv069dPe/fuldPpLF23e/duhYSEyMfHp4Kx4el6tmioV4Z1l8Vy9im/c9emmB0JAFBFXL5MExsbq7lz52rhwoVKTk7WuHHjlJeXp9GjR0uSRo4cqbi4uNLtx40bp5MnT+qhhx7S7t27tXz5ck2dOlXjx4+vvHcBjzSoa4ieGHSJJGnqip369PujJicCAFQFL1d3GDZsmI4fP67JkycrLS1N3bt318qVK0sHtaampspq/aXjhIWF6YsvvtCECRPUrVs3NWvWTA899JAee+yxynsX8Fh3D2ito6cLNH/9fv1jyfdqUt+u6DaNzY4FAKhELs8zYgbmGandnE5DD3ywRSu2pcnP10sfjeur9kHMQQIANV2VzDMCmMFqtWj60O7qHd5QOQUlGjV/k9KymIMEADwFZQRuwdfbprkje6lNk3o6llWgO9/apOyCYrNjAQAqAWUEbqNBXR8tGN1HTfzs2pmWo3HvJqqoxHnhHQEANRplBG4lrFFdvXVnb9XzsWn93hN67KMfmIMEANwcZQRup0uzAL1xR6RsVouWbT2iF77YZXYkAMBFoIzALV3evomm3dxVkjRr9T69tX6/yYkAABVFGYHbGtorTP+4pr0k6V+f72BSNABwU5QRuLXxV7bVqOiWMgzp70uStG5PptmRAAAuoozArVksFk0e3FnXdwtRscPQ2Hc2a/uRLLNjAQBcQBmB27NZLZo+NEJ92zRWXpFDd761SQcy88yOBQAoJ8oIPILdy6b/jIhU51B/ZeYWaeT8TcrIYZZWAHAHlBF4DD9fb701urdaNKqr1JP5unP+d8phllYAqPEoI/AoTf189fZdfRRY30c7jmXr3rcTVVjiMDsWAOAPUEbgccID62nB6D6q52NTQsoJTVicJIeTWVoBoKaijMAjdWkWoDkje8nbZtGKbWma/H/bmTYeAGooygg8Vr+2gXplWHdZLNJ736bq5S93mx0JAHAOlBF4tBu6heqZIV0kSTO+2au5a1JMTgQA+C3KCDze7VEt9ei1HSRJz65I1pLvDpmcCADwa5QR1ArjLm+jsZe1liRN/PgH/XfbMZMTAQB+RhlBrWCxWDTxuo4a1itMTkN6aFGS1u45bnYsAIAoI6hFLBaLpt7cVYO6BqvI4dTYdxK1JfWU2bEAoNajjKBWsVktemVYdw1oF6j8IodGv/WddqZlmx0LAGo1yghqnZ+fY9OzRQNlnSnWiHmblHoi3+xYAFBrUUZQK9X18dJbd/ZRx2A/Hc8p1O3zNio9mwfrAYAZKCOotQLqeuvtu/qoZeO6OnTyjO5481udyC00OxYA1DqUEdRqTf199e6YKAX7+2pPRq5Gzt+krDM86RcAqhNlBLVeWKO6eu+eKAXW99GPR7N151ublFtYYnYsAKg1KCOApDZN6uudMVEKqOOtramnNWbBdzpT5DA7FgDUCpQR4CeXhPjrnTF95Gf30rf7T2rsu4kqLKGQAEBVo4wAv9KteQO9Nbq36njbtGb3cT34/lYVO5xmxwIAj0YZAX6jV3gjvTmql3y8rPpyR7pil3wvh9MwOxYAeCzKCHAO/doGavYdPeVts+iz748q7uMf5KSQAECVoIwA53FVxyD9+689ZLVISzYf1lOf/SjDoJAAQGWjjAB/YFDXEL08NEIWi7Qw4aCe++9OCgkAVDLKCHABN/VormeHdJUk/WdNil7+cjeFBAAqEWUEKIfhUS00ZXAnSdKMb/bqla/2mJwIADwHZQQop9H9WunJ6y+RJL0Wv0f/ppAAQKWgjAAuuHtAaz0+qKMk6ZWvdmvmN3tNTgQA7o8yArjo3sva6LFrzxaSF7/YpVmr95mcCADcG2UEqIBxV7TRIwM7SJKeX7lTc9ZQSACgoigjQAWNv7KtYq9uL0maumKn3lybYnIiAHBPlBHgIvztT+300J/aSZKeWZ6s+ev2m5wIANwPZQS4SA/HtNMDV7aVJP3r8x1auOGAuYEAwM1QRoCLZLFY9Pdr2mvcFW0kSVM+/VHvbDxocioAcB+UEaASWCwWPTqwg8Ze1lqSNOmT7Vqwnks2AFAelBGgklgsFk28rmNpIfnnZzsY1AoA5UAZASrRz4Xk5zEkzyxPZh4SALgAyghQyX4eQ/JwzNm7bJ5fuVOvxTN1PACcD2UEqAIWi0UPx7QvnRht+qrdmv7lLp72CwDnQBkBqtD4K9sq7rqzU8e/9vVevfAFhQQAfosyAlSxsZe30aQbOkmSZq3ep6krkikkAPArlBGgGozp30r/+nNnSdLctfv11Gc7KCQA8BPKCFBNRkaHa+pNXSVJCzYc0KT/2y6nk0ICAJQRoBoNj2qhF27pJotFendjquI+3iYHhQRALUcZAarZ0F5hmj40QlaLtHjzIT28OEnFDqfZsQDANJQRwAQ39Wiu127rIS+rRZ99f1Tj3k1UQbHD7FgAYIoKlZGZM2cqPDxcvr6+ioqK0qZNm8q136JFi2SxWDRkyJCKvCzgUW7oFqo5IyNl97Lqq+QM3bXgO+UVlpgdCwCqnctlZPHixYqNjdWUKVO0ZcsWRUREaODAgcrIyPjD/Q4cOKB//OMfGjBgQIXDAp7mqo5BWjC6j+r52LRh3wndMe9bZeUXmx0LAKqVy2Vk+vTpuueeezR69Gh16tRJs2fPVt26dTV//vzz7uNwOHT77bfrqaeeUuvWrS8qMOBpots01nv3XKqAOt7amnpaf527UZm5hWbHAoBq41IZKSoqUmJiomJiYn75BlarYmJilJCQcN79/vWvf6lp06YaM2ZMuV6nsLBQ2dnZZRbAk3UPa6DFYy9VYH27ko9la+jsBB09fcbsWABQLVwqI5mZmXI4HAoKCiqzPigoSGlpaefcZ926dZo3b57mzp1b7teZNm2aAgICSpewsDBXYgJuqWOwv5beF61mDeooJTNPt85O0IHMPLNjAUCVq9K7aXJycjRixAjNnTtXgYGB5d4vLi5OWVlZpcuhQ4eqMCVQc7QKrKel90WrdWA9HTl9Rrf+J0G70nLMjgUAVcrLlY0DAwNls9mUnp5eZn16erqCg4N/t/2+fft04MABDR48uHSd03l2PgUvLy/t2rVLbdq0+d1+drtddrvdlWiAxwhtUEeLx0Zr5PxNSj6WrWFzErRwdB9FhDUwOxoAVAmXzoz4+PgoMjJS8fHxpeucTqfi4+MVHR39u+07duyobdu2KSkpqXS58cYbdeWVVyopKYnLL8B5NPGza9E9l6pHiwY6nV+s4XM3KmHfCbNjAUCVcPkyTWxsrObOnauFCxcqOTlZ48aNU15enkaPHi1JGjlypOLi4iRJvr6+6tKlS5mlQYMG8vPzU5cuXeTj41O57wbwIAF1vfXumCj1bdNYeUUOjZq/SSu3HzM7FgBUOpfLyLBhw/TSSy9p8uTJ6t69u5KSkrRy5crSQa2pqak6doxfmEBlqGf30vw7e+vazsEqcjh1/3tb9P63qWbHAoBKZTHc4Dnm2dnZCggIUFZWlvz9/c2OA1Q7h9PQk59s1webzhaR2Kvb68Gr2spisZicDADOr7yf3zybBnADNqtFU2/qor9d1VaSNH3Vbk359Eee+AvAI1BGADdhsVgUe00HPXVjZ1ks0tsJB/W3RVtVWMID9gC4N8oI4GZG9Q3X67f1kLfNouU/HNNdC75TLg/YA+DGKCOAG7qhW6jeuvPsA/bW7z2hv85J4Hk2ANwWZQRwU/3bBeqDey9V43o+2n4kW7fM2qBDJ/PNjgUALqOMAG6sW/MGWnpftJo3rKMDJ/J186wN2nGUB0sCcC+UEcDNtW5SXx+N66uOwX46nlOoYf9J0Ia9mWbHAoByo4wAHiDI31eLx0arT3gj5RSWaNRbm7Rs62GzYwFAuVBGAA8RUMdbb4/po+u7hajYYWjC4u/1evweucG8hgBqOcoI4EF8vW16/a89NPay1pKkl1ft1sSPtqnY4TQ5GQCcH2UE8DBWq0Vxgy7R03/uLKtFWrz5kMYs3MxcJABqLMoI4KFGRIdrzohequNt05rdxzV0doLSswvMjgUAv0MZATxYTKcgLR57qQLr+2jHsWzdNHO9dqXlmB0LAMqgjAAerlvzBlp2fz+1aVJPR7MKdMusDVrPrb8AahDKCFALhDWqq4/G9VWfVj/d+jt/kz5K5NZfADUDZQSoJRrU9dE7Y/pocESoSpyG/r70e/37K279BWA+yghQi9i9bPr3sO667/I2kqRXvtqtCYuTVFDsMDkZgNqMMgLUMlarRROv66hnb+oim9WiT5KOavjcjTqew1N/AZiDMgLUUrdHtdTbd/WRv6+XtqSe1pCZ65V8jIfsAah+lBGgFuvXNlCfjO+nVoH1dOT0Gd0ya4O+2pFudiwAtQxlBKjlWjepr2X391XfNo2VV+TQPe9s1tw1KQxsBVBtKCMA1KCujxbe1UfDo1rIMKRnVyTrsY9+UFEJz7QBUPUoIwAkSd42q54d0kVTBneS1SIt2XxYI+Z9q1N5RWZHA+DhKCMASlksFo3u10rz7uyt+nYvfbv/pIa8sV57M5hCHkDVoYwA+J0rOzTVx/f3VVijOjp4Il83vbFBa3YfNzsWAA9FGQFwTu2D/PTJ/f3UO7yhcgpKdOdbm/TmWga2Aqh8lBEA59W4vl3v3h2lWyOby2lIzyxP1oTFSTpTxIytACoPZQTAH7J72fTCLd30z8GdSmdsvWX2Bh0+lW92NAAegjIC4IIsFovu7NdK790dpcb1fPTj0WwNfn2dNuzNNDsaAA9AGQFQbpe2bqxPH+yvrs0CdCq/WCPmM44EwMWjjABwSbMGdbT0vmjd3LOZHE6jdBwJT/4FUFGUEQAu8/W26eVbIzTlV+NI/jKLcSQAKoYyAqBCfp4g7d0xUWr00ziSG2es14Z9jCMB4BrKCICLEt2msT57sL+6NPPXybwijZi3SfPW7WccCYByo4wAuGjNGtTRh/f11U09zo4jefrzHfrboiTlFZaYHQ2AG6CMAKgUvt42TR8aock3nB1H8tn3R3XjjHXanc5zbQD8McoIgEpjsVh0V/9WWnzvpQr299W+43n684z1Wrb1sNnRANRglBEAla5XeCMt/1t/DWgXqDPFDk1Y/L3iPt7G7b8AzokyAqBKNK5v14LRffTQn9rJYpE+2JSqv8zaoNQT3P4LoCzKCIAqY7NaNOHq9lo4uo8a1vXWj0ezdf3ra/Xlj2lmRwNQg1BGAFS5y9o30fK/DVDPFg2UU1Cie99J1LQVySp2OM2OBqAGoIwAqBahDepo8dhojenfSpL0nzUpGj53o9KzC0xOBsBslBEA1cbbZtWkGzpp1u095Wf30ncHTun619Zq3R5mbQVqM8oIgGp3XdcQffpgf3UM9lNmbpFGzP9Wz6/cyWUboJaijAAwRavAevpkfD/d1qeFDEOatXqfbpmdwN02QC1EGQFgGl9vm6bd3FWzbu8pf18vfX/otAa9tlb/l3TE7GgAqhFlBIDprusaov8+fJl6hzdUbmGJHlqUpH8s/Z5n2wC1BGUEQI3QrEEdfXDPpXroT+1ktUgfJh7WDa+v0/YjWWZHA1DFKCMAagwvm1UTrm6vD+65VCEBvtqfmaeb3livN9emyOk0zI4HoIpQRgDUOFGtG+u/Dw3QwM5BKnYYemZ5su5a+J0ycwvNjgagClBGANRIDer6aPYdkXpmSBfZvaxaveu4rn11rdbuOW52NACVjDICoMayWCy649KW+vSB/mofVF+ZuYUaMW+T/vnpjzpTxBOAAU9BGQFQ43UI9tOnD/TXiEtbSpIWbDig619fq+8PnTY3GIBKQRkB4BZ8vW16ekgXLbyrj4L87Uo5nqebZ23QK6t2M3Mr4OYoIwDcyuXtm+iLhy/T4IhQOZyG/h2/Rze/sUF7M3LMjgaggigjANxOg7o+ev22Hnrtth4KqOOtbUeydP1r6zR/3X5uAQbcUIXKyMyZMxUeHi5fX19FRUVp06ZN59127ty5GjBggBo2bKiGDRsqJibmD7cHgPK6MSJUX064TJe1b6LCEqf+9fkO3THvWx05fcbsaABc4HIZWbx4sWJjYzVlyhRt2bJFERERGjhwoDIyMs65/erVq3Xbbbfpm2++UUJCgsLCwnTNNdfoyBGePQHg4gX5+2rh6N56ekgX1fG2acO+E7r2lTX6KPGwDIOzJIA7sBgu/q81KipKvXv31owZMyRJTqdTYWFhevDBBzVx4sQL7u9wONSwYUPNmDFDI0eOLNdrZmdnKyAgQFlZWfL393clLoBaZH9mnmKXJGlr6mlJ0rWdg/X0kC5q4mc3NxhQS5X389ulMyNFRUVKTExUTEzML9/AalVMTIwSEhLK9T3y8/NVXFysRo0anXebwsJCZWdnl1kA4EJaBdbT0rHRemRgB3lZLVr5Y5qufuV/+ngLZ0mAmsylMpKZmSmHw6GgoKAy64OCgpSWllau7/HYY48pNDS0TKH5rWnTpikgIKB0CQsLcyUmgFrMy2bV+Cvb6v8e6KdOIf46nV+s2CXfa/SC73SUsSRAjVStd9M899xzWrRokZYtWyZfX9/zbhcXF6esrKzS5dChQ9WYEoAn6BwaoP97oJ8eGdhBPraz08lf88oavfftQe64AWoYl8pIYGCgbDab0tPTy6xPT09XcHDwH+770ksv6bnnntOXX36pbt26/eG2drtd/v7+ZRYAcJX3T2dJVjzUXz1bNFBuYYmeWLZdw9/cqAOZeWbHA/ATl8qIj4+PIiMjFR8fX7rO6XQqPj5e0dHR593vhRde0NNPP62VK1eqV69eFU8LABXQtqmflt7XV5Nv6KQ63jZtTDmpa/+9RnPXpMjBWRLAdC5fpomNjdXcuXO1cOFCJScna9y4ccrLy9Po0aMlSSNHjlRcXFzp9s8//7wmTZqk+fPnKzw8XGlpaUpLS1Nubm7lvQsAuACb1aK7+rfSFw9fpr5tGqug2KlnVyTrL7M2aHc6s7cCZnK5jAwbNkwvvfSSJk+erO7duyspKUkrV64sHdSampqqY8eOlW4/a9YsFRUV6ZZbblFISEjp8tJLL1XeuwCAcmrRuK7euztKz93cVX52LyUdOq3rX1ur1+L3qKiEZ9wAZnB5nhEzMM8IgKpwLOuMnly2XfE7z07a2LZpfT0zpIsubd3Y5GSAZ6iSeUYAwJOEBNTRm6N66d9/7a7A+j7am5Grv87ZqNglScrMLTQ7HlBrUEYA1GoWi0V/7t5M8bFX6PaoFrJYpI+3HNGfXv6fPtiUym3AQDXgMg0A/MqW1FN6Ytl2JR87O/NzzxYN9OxNXXVJCL97AFdxmQYAKqBni4b67IF+mnRDJ9XzsWlL6mnd8Po6Pbt8h/IKS8yOB3gkyggA/IaXzaox/Vvpq79fruu6BMvhNDR37X7FTP+fvvgxjefcAJWMMgIA5xESUEez7ojUW3f2VvOGdXQsq0Bj30nU3Qs36+AJZnAFKgtlBAAu4MqOTbVqwuUaf2Ubedssit+Zoaunr9HzK3cql0s3wEVjACsAuGBvRo6e+myH1u7JlCQ19bPrsWs76qYezWS1WkxOB9Qs5f38powAgIsMw9BXyRl6ZvkOHTyRL0nqHtZAUwZ3Uo8WDU1OB9QclBEAqGKFJQ7NX3dAM77eo7wihyTpLz2b67FrO6ipv6/J6QDzUUYAoJpkZBfohS926cPEw5Kkej42PXBVO93VP1x2L5vJ6QDzUEYAoJolHTqtf376o5IOnZYktWxcV48PukTXdAqSxcJ4EtQ+lBEAMIHTaeiTpCN67r87lZFz9vk2vcMbKm7QJerJeBLUMpQRADBRbmGJZq3eqzfX7ldhiVOSdF2XYD0ysINaN6lvcjqgelBGAKAGOJZ1Rq+s2q0PEw/LaUheVotu69NCf/tTOzXxs5sdD6hSlBEAqEF2peXo+ZU79fXODElnB7mOvbyN7h7QSnV9vExOB1QNyggA1EAJ+05o2n+T9cPhLElSEz+7JsS019BezeVlY1JseBbKCADUUE6noeXbjumFL3bq0MkzkqQ2Terp79d00LWdg5nJFR6DMgIANVxRiVPvfXtQr8Xv0an8YknSJSH+mhDTTldzOzA8AGUEANxEdkGx3ly7X/PX7S998F635gGacHV7XdG+CaUEbosyAgBu5lRekeasTdGC9Qd0pvjs9PKRLRsq9ur26tumMaUEbocyAgBuKjO3UP/53z69nXCwdI6SqFaN9PdrOqhPq0YmpwPKjzICAG4uI7tAb6zep/e/TVWR42wpGdAuUA/9qZ16hVNKUPNRRgDAQxw9fUYzvtmrJd8dUonz7K/sPq0a6YEr22pAu0Au36DGoowAgIc5dDJfM7/Zq4+2HFax4+yv7m7NA3T/FW11TacgbglGjUMZAQAPdSzrjOau2a/3Nx1UQfHZyzftmtbX/Ve20eBuoUyehhqDMgIAHu5EbqHeWn9ACxMOKKfg7C3BYY3q6L7L2+gvPZvL19tmckLUdpQRAKglsguK9U7CQc1ft18n8ookSU397LqzX7iG92mhBnV9TE6I2ooyAgC1zJkihxZ/l6r/rEnRsawCSVIdb5uG9mqu0f1aKTywnskJUdtQRgCglioqceqz749q7toU7UzLkSRZLNLVlwTp7gGt1Tu8IXfgoFpQRgCgljMMQxv2ndCba1P0za7jpesjmgdozIDWuq5LsLwZ7IoqRBkBAJTam5Gjeev266MtR1T006yuoQG+Gtk3XEN7halRPcaVoPJRRgAAv5OZW6j3NqbqnY0HlJl7drCrj5dVN3QL0cjocHUPa2BuQHgUyggA4LwKih36NOmo3t54QNuPZJeu79Y8QHdc2lI3RoRyazAuGmUEAHBBhmEo6dBpvZNwUJ//cKz0GTgBdbw1tFdz3R7VkrtwUGGUEQCAS07kFmrJ5sN6d+NBHTl9pnT9gHaBGtY7TFd3CpLdi7MlKD/KCACgQhxOQ6t3ZeidjQf1v93H9fOnRMO63hrSo5mG9Q5Tx2B+F+PCKCMAgIuWeiJfSzYf0oeJh5WWXVC6PqJ5gIb2DtPgiFD5+3qbmBA1GWUEAFBpHE5Da3Yf1+LvDumr5HSVOM9+dPh6WzWoS4huiWyuqNaNZePJwfgVyggAoEpk5hZq2ZYjWrz5kPZm5JauD/b31Y3dQzWkezNdEuLHLK+gjAAAqpZhGNp66LSWbj6s5T8cVfZPTw6WpA5BfhrSo5n+3D1UoQ3qmJgSZqKMAACqTWGJQ9/sPK7/Szqi+OSM0luEJSmqVSPd2D1U13YOVuP6dhNTorpRRgAApsg6U6z/bjumT5KOaGPKydL1VosU3aaxBnUN0cDOwQqkmHg8yggAwHRHTp/Rp0lHtWLbMW07klW63mqRLm19tphc24Vi4qkoIwCAGiX1RL5WbD+mFduO6YfDZYtJr5aNdHWnIMV0ClIrZnz1GJQRAECNdb5iIkltm9ZXzCVBurpTkHqENZCV24XdFmUEAOAWDp/K11c70vVVcoY2ppwoncNEkgLr++jKDk11RYem6t8uUAF1mGDNnVBGAABuJ+tMsVbvytBXyRlavTNDOYW/3C5ss1rUI6yBrujQRFd0aKpOIf6cNanhKCMAALdWVOLUpv0n9c2uDK3elaF9x/PK/HtgfbsGtAtU3zaN1a9tIPOZ1ECUEQCARzl0Ml//231c/9t9XBv2ZiqvyFHm38Mb11XftoHq1yZQ0W0aq1E9H5OS4meUEQCAxyoqcWrzwZNavzdT6/ee0A+HT8v5m0+zdk3rq1d4Q0W2bKReLRuqZeO6TFFfzSgjAIBaI7ugWN+mnNSGfZnasPeEdqXn/G6bwPo+imzZUL1aNlJkeEN1CQ2Qj5fVhLS1B2UEAFBrncgtVOLBU0o8eEqbD57StsNZZaaolyS7l1URzRsoMryhIls0VNfmAWrqZ+fsSSWijAAA8JOCYoe2H8nS5oOntPnAKSUePKlT+cW/2y6wvl1dmvmrc6i/uoQGqEuzADVvWIeCUkGUEQAAzsMwDKVk5inxwCl9d+Ckkg6d1r7jub8bdyJJ/r5e6hwa8FNJCVD7ID+1blJPvt626g/uZigjAAC44EyRQzvTsrX9aLZ+PJKl7UeztCstR8WO339MWi1Si0Z11bapn9o2ra92TeurXVB9tWlSX/XsXiakr5mqtIzMnDlTL774otLS0hQREaHXX39dffr0Oe/2S5cu1aRJk3TgwAG1a9dOzz//vAYNGlTu16OMAADMUFTi1O70HP14NEvbj2Rrx7Fs7UnPUXZByXn3CQ3wVYvGddWyUb2zX3/159o2g2yVlZHFixdr5MiRmj17tqKiovTqq69q6dKl2rVrl5o2bfq77Tds2KDLLrtM06ZN0w033KD3339fzz//vLZs2aIuXbpU6psBAKCqGYah47mF2pueq73Hc7UnPVd7MnK0NyNXmblFf7hvg7reCg2oo5AAXwUH+P709Ze/B/v7etSZlSorI1FRUerdu7dmzJghSXI6nQoLC9ODDz6oiRMn/m77YcOGKS8vT59//nnpuksvvVTdu3fX7NmzK/XNAABgplN5RUrJzFPqyTwdPJGv1BP5OngyXwdP5Cszt7Bc38PP10tN/OxqVNdHDev5/PK1nrca1vVR4/o+aljXR36+3qpv91I9u011fbxkq4FT45f389ul+lVUVKTExETFxcWVrrNarYqJiVFCQsI590lISFBsbGyZdQMHDtQnn3xy3tcpLCxUYeEv/9Gys7NdiQkAgCka1vNRZL2z85n8Vl5hiQ6dytex0wU6llWgtOwCpWWdOfvnn5acwhLlFJxdUpR3jlc4vzreNtWze6m+/ezXej5e8vayyMtqlbfNKm+b5aevv/z51wVmTP9WCmtU96KPQUW4VEYyMzPlcDgUFBRUZn1QUJB27tx5zn3S0tLOuX1aWtp5X2fatGl66qmnXIkGAECNVs/upY7B/uoYfP4zBDkFxUrPLlBmbpFO5RXpZH6RTuae/Xr278U6mVeoU3nFyikoVl6RQ46fbgE6U+zQmWKHMnMrlu/G7qHuUUaqS1xcXJmzKdnZ2QoLCzMxEQAAVc/P11t+vt5q+/shmOdkGIYKS5zKLSxRXmGJ8godyisqUW5hic4UOVTscKrYYajY4VSJw6kih6ESh1PFP/3ZMAwZhmTIULC/b9W+uT/gUhkJDAyUzWZTenp6mfXp6ekKDg4+5z7BwcEubS9JdrtddrvdlWgAANQ6FotFvt42+XrbFFjffT83XZqU38fHR5GRkYqPjy9d53Q6FR8fr+jo6HPuEx0dXWZ7SVq1atV5twcAALWLy5dpYmNjNWrUKPXq1Ut9+vTRq6++qry8PI0ePVqSNHLkSDVr1kzTpk2TJD300EO6/PLL9fLLL+v666/XokWLtHnzZs2ZM6dy3wkAAHBLLpeRYcOG6fjx45o8ebLS0tLUvXt3rVy5snSQampqqqzWX0649O3bV++//76efPJJPf7442rXrp0++eSTcs8xAgAAPBvTwQMAgCpR3s9vl8aMAAAAVDbKCAAAMBVlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqSgjAADAVJQRAABgKpengzfDz5PEZmdnm5wEAACU18+f2xea7N0tykhOTo4kKSwszOQkAADAVTk5OQoICDjvv7vFs2mcTqeOHj0qPz8/WSyWSvu+2dnZCgsL06FDh3jmTRXiOFcfjnX14DhXD45z9ajK42wYhnJychQaGlrmIbq/5RZnRqxWq5o3b15l39/f358f9GrAca4+HOvqwXGuHhzn6lFVx/mPzoj8jAGsAADAVJQRAABgqlpdRux2u6ZMmSK73W52FI/Gca4+HOvqwXGuHhzn6lETjrNbDGAFAACeq1afGQEAAOajjAAAAFNRRgAAgKkoIwAAwFQeX0Zmzpyp8PBw+fr6KioqSps2bfrD7ZcuXaqOHTvK19dXXbt21YoVK6opqXtz5TjPnTtXAwYMUMOGDdWwYUPFxMRc8L8LfuHqz/TPFi1aJIvFoiFDhlRtQA/h6nE+ffq0xo8fr5CQENntdrVv357fH+Xg6nF+9dVX1aFDB9WpU0dhYWGaMGGCCgoKqimte1qzZo0GDx6s0NBQWSwWffLJJxfcZ/Xq1erZs6fsdrvatm2rBQsWVG1Iw4MtWrTI8PHxMebPn2/8+OOPxj333GM0aNDASE9PP+f269evN2w2m/HCCy8YO3bsMJ588knD29vb2LZtWzUndy+uHufhw4cbM2fONLZu3WokJycbd955pxEQEGAcPny4mpO7H1eP9c/2799vNGvWzBgwYIDx5z//uXrCujFXj3NhYaHRq1cvY9CgQca6deuM/fv3G6tXrzaSkpKqObl7cfU4v/fee4bdbjfee+89Y//+/cYXX3xhhISEGBMmTKjm5O5lxYoVxhNPPGF8/PHHhiRj2bJlf7h9SkqKUbduXSM2NtbYsWOH8frrrxs2m81YuXJllWX06DLSp08fY/z48aV/dzgcRmhoqDFt2rRzbj906FDj+uuvL7MuKirKGDt2bJXmdHeuHuffKikpMfz8/IyFCxdWVUSPUZFjXVJSYvTt29d48803jVGjRlFGysHV4zxr1iyjdevWRlFRUXVF9AiuHufx48cbV111VZl1sbGxRr9+/ao0pycpTxl59NFHjc6dO5dZN2zYMGPgwIFVlstjL9MUFRUpMTFRMTExpeusVqtiYmKUkJBwzn0SEhLKbC9JAwcOPO/2qNhx/q38/HwVFxerUaNGVRXTI1T0WP/rX/9S06ZNNWbMmOqI6fYqcpw//fRTRUdHa/z48QoKClKXLl00depUORyO6ortdipynPv27avExMTSSzkpKSlasWKFBg0aVC2ZawszPgvd4kF5FZGZmSmHw6GgoKAy64OCgrRz585z7pOWlnbO7dPS0qosp7uryHH+rccee0yhoaG/++FHWRU51uvWrdO8efOUlJRUDQk9Q0WOc0pKir7++mvdfvvtWrFihfbu3av7779fxcXFmjJlSnXEdjsVOc7Dhw9XZmam+vfvL8MwVFJSovvuu0+PP/54dUSuNc73WZidna0zZ86oTp06lf6aHntmBO7hueee06JFi7Rs2TL5+vqaHcej5OTkaMSIEZo7d64CAwPNjuPRnE6nmjZtqjlz5igyMlLDhg3TE088odmzZ5sdzaOsXr1aU6dO1RtvvKEtW7bo448/1vLly/X000+bHQ0XyWPPjAQGBspmsyk9Pb3M+vT0dAUHB59zn+DgYJe2R8WO889eeuklPffcc/rqq6/UrVu3qozpEVw91vv27dOBAwc0ePDg0nVOp1OS5OXlpV27dqlNmzZVG9oNVeRnOiQkRN7e3rLZbKXrLrnkEqWlpamoqEg+Pj5VmtkdVeQ4T5o0SSNGjNDdd98tSeratavy8vJ077336oknnpDVyv+/rgzn+yz09/evkrMikgefGfHx8VFkZKTi4+NL1zmdTsXHxys6Ovqc+0RHR5fZXpJWrVp13u1RseMsSS+88IKefvpprVy5Ur169aqOqG7P1WPdsWNHbdu2TUlJSaXLjTfeqCuvvFJJSUkKCwurzvhuoyI/0/369dPevXtLy54k7d69WyEhIRSR86jIcc7Pz/9d4fi5ABo8Zq3SmPJZWGVDY2uARYsWGXa73ViwYIGxY8cO49577zUaNGhgpKWlGYZhGCNGjDAmTpxYuv369esNLy8v46WXXjKSk5ONKVOmcGtvObh6nJ977jnDx8fH+PDDD41jx46VLjk5OWa9Bbfh6rH+Le6mKR9Xj3Nqaqrh5+dnPPDAA8auXbuMzz//3GjatKnxzDPPmPUW3IKrx3nKlCmGn5+f8cEHHxgpKSnGl19+abRp08YYOnSoWW/BLeTk5Bhbt241tm7dakgypk+fbmzdutU4ePCgYRiGMXHiRGPEiBGl2/98a+8jjzxiJCcnGzNnzuTW3ov1+uuvGy1atDB8fHyMPn36GBs3biz9t8svv9wYNWpUme2XLFlitG/f3vDx8TE6d+5sLF++vJoTuydXjnPLli0NSb9bpkyZUv3B3ZCrP9O/RhkpP1eP84YNG4yoqCjDbrcbrVu3Np599lmjpKSkmlO7H1eOc3FxsfHPf/7TaNOmjeHr62uEhYUZ999/v3Hq1KnqD+5Gvvnmm3P+zv352I4aNcq4/PLLf7dP9+7dDR8fH6N169bGW2+9VaUZLYbBuS0AAGAejx0zAgAA3ANlBAAAmIoyAgAATEUZAQAApqKMAAAAU1FGAACAqSgjAADAVJQRAABgKsoIAAAwFWUEAACYijICAABMRRkBAACm+n8RRhIeZqqUIAAAAABJRU5ErkJggg==\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mse = lambda pred, actual: (actual - pred) ** 2\n",
    "output_probs = sigmoid(nn_outputs)\n",
    "\n",
    "plt.plot(output_probs, mse(output_probs, 1))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see, mean squared error only ranges between `0` and `1`.  This can make it hard for the network to penalize incorrect guesses and adjust the weights properly.  We instead want a function that will penalize incorrect predictions much more than predictions that are close to the actual value.\n",
    "\n",
    "We can instead use the negative log likelihood, which is defined as $NLL = -(y * log(\\hat{y}) + (1-y) * log(1-\\hat{y}))$.  Since $y$ is always `0` or `1`, one half of the equation will always be `0`.  Let's say $y$ is `1`, and $\\hat{y}$ is `.7`.\n",
    "\n",
    "$$NLL = -(1 * log(.7) + (1-1) * (1-.7))$$\n",
    "$$NLL = -(1 * log(.7) + 0 * (1-.7))$$\n",
    "$$NLL = -(1 * log(.7))$$\n",
    "$$NLL = -(1 * log(.7))$$\n",
    "$$NLL = .3566$$\n",
    "\n",
    "Remember that $log .7$ means \"to what power do I need to raise $e$ to get `.7`?\"  To get a number between `0` and  `1` (which is what the sigmoid function outputs), you have to raise $e$ to a negative power.  Raising to a negative power is the same as repeatedly multiplying $1/e$ by itself.  The closer $\\hat{y}$ is to `1`, the lower the negative power (the further from `0`).\n",
    "\n",
    "You then \"undo\" the negative with the beginning negative sign for negative log likelihood.  Note that $log 0$ is undefined, so we have to be careful not to pass `0` in as our prediction.  Luckily, the sigmoid function cannot output `0`, since the sigmoid is $1/(1+e^{-x}$.  As $e^{-x}$ increases, the sigmoid outputs numbers closer and closer to `0`, but never reaches `0`.\n",
    "\n",
    "Let's see how negative log likelihood changes as we change our predictions.  We're assuming that the correct target is `1`:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "outputs": [
    {
     "data": {
      "text/plain": "[<matplotlib.lines.Line2D at 0x177bf92a0>]"
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAwwElEQVR4nO3deXhb5YH3/Z8kW5I3ybtjx8q+kQUChKQJW6EpKaUUZt6ZwpQyKTNTmJJCS/q2DaVspRDK8HbywGSgZVqgzwtkaIftGQKUpgQmELYsNM2+x4ljx7u8SrZ0nj8kK3biLHYkHcnn+7kuXY6lY+v2GRd/5z73OcdmGIYhAACAJLGbPQAAAGAtxAcAAEgq4gMAACQV8QEAAJKK+AAAAElFfAAAgKQiPgAAQFIRHwAAIKkyzB7AscLhsKqrq5WXlyebzWb2cAAAwGkwDEOtra2qqKiQ3X7yuY2Ui4/q6mr5fD6zhwEAAIagqqpKlZWVJ90m5eIjLy9PUmTwHo/H5NEAAIDT4ff75fP5Yn/HTybl4qP3UIvH4yE+AABIM6ezZIIFpwAAIKmIDwAAkFTEBwAASCriAwAAJBXxAQAAkor4AAAASUV8AACApCI+AABAUhEfAAAgqQYdH++9956uvvpqVVRUyGaz6ZVXXun3umEYuueee1ReXq6srCzNnz9fO3fujNd4AQBAmht0fLS3t+ucc87R8uXLB3z9kUce0WOPPaYnn3xSH330kXJycrRgwQJ1dXWd8WABAED6G/S9Xa688kpdeeWVA75mGIaWLVumn/zkJ7rmmmskSb/97W9VVlamV155Rddff/2ZjRYAAKS9uK752Lt3r2pqajR//vzYc16vV3PmzNHatWsH/JpAICC/39/vkQh1rQHd99pmPfzGtoR8fwAAcHriGh81NTWSpLKysn7Pl5WVxV471tKlS+X1emMPn88XzyHF+Lu69cwH+/T8R/sT8v0BAMDpMf1slzvvvFMtLS2xR1VVVULfz0jodwcAAKcS1/gYMWKEJKm2trbf87W1tbHXjuVyueTxePo9EsGWkO8KAAAGK67xMXbsWI0YMUKrVq2KPef3+/XRRx9p7ty58XyroWPqAwAAUw36bJe2tjbt2rUr9vnevXu1ceNGFRYWatSoUfre976nn/3sZ5o4caLGjh2ru+++WxUVFbr22mvjOe5Bs9mY+wAAIBUMOj4+/fRTXXbZZbHPFy9eLElauHChnnnmGf3whz9Ue3u7br75ZjU3N+uiiy7Sm2++KbfbHb9RnwEmPgAAMJfNMIyU+nvs9/vl9XrV0tIS1/Uf++rb9flHVyvH6dDmn34pbt8XAAAM7u+36We7JAtHXQAASA2WiY9eKTXNAwCABVkmPmycbAsAQEqwTHz0Sq0VLgAAWI9l4oM1HwAApAbLxEcvg1UfAACYynLxAQAAzGW5+GDNBwAA5rJMfLDmAwCA1GCZ+OjFxAcAAOayTHxwYzkAAFKDZeIjhqkPAABMZZn4YN4DAIDUYJn46MV1PgAAMJdl4oMlHwAApAbLxEcvrvMBAIC5LBMfvXe1pT0AADCXdeKDwy4AAKQEy8RHL4PjLgAAmMoy8cHEBwAAqcEy8dGLeQ8AAMxlnfhg6gMAgJRgnfiIYskHAADmskx82Jj6AAAgJVgmPgAAQGqwTHxwnQ8AAFKDZeKjL671AQCAeSwTH0x8AACQGiwTH30x8QEAgHksEx82Fn0AAJASLBMffTHxAQCAeSwTH33nPVhwCgCAeSwTHwAAIDVYJj76Lvlg3gMAAPNYJz442RYAgJRgmfjoiyUfAACYxzrxwcQHAAApwTrx0YfBqg8AAExjmfjgGmMAAKQGy8RHX6z5AADAPJaJDyY+AABIDZaJDwAAkBosEx/cWA4AgNRgmfjoizUfAACYxzLxwbwHAACpwTLx0RfX+QAAwDyWiQ+WfAAAkBosEx99seYDAADzWCY++t7VlvYAAMA81okPDrsAAJASLBMffRkcdwEAwDSWjA8AAGAeS8YH8x4AAJjHMvHBmg8AAFKDZeKjL5Z8AABgHsvEh40LrAMAkBIsEx/9MPMBAIBp4h4foVBId999t8aOHausrCyNHz9eDzzwgOmnt7LmAwCA1JAR72/485//XE888YSeffZZTZs2TZ9++qluuukmeb1e3X777fF+uyHhxnIAAJgn7vHxwQcf6JprrtFVV10lSRozZoxeeOEFffzxx/F+q0Fh4gMAgNQQ98Mu8+bN06pVq7Rjxw5J0meffaY1a9boyiuvHHD7QCAgv9/f75FonO0CAIB54j7zsWTJEvn9fk2ZMkUOh0OhUEgPPvigbrjhhgG3X7p0qe6///54D+M4NhZ9AACQEuI+8/Hiiy/queee0/PPP6/169fr2Wef1aOPPqpnn312wO3vvPNOtbS0xB5VVVXxHtJxmPgAAMA8cZ/5+MEPfqAlS5bo+uuvlyTNmDFD+/fv19KlS7Vw4cLjtne5XHK5XPEexnGY9wAAIDXEfeajo6NDdnv/b+twOBQOh+P9VoPS96hLmEUfAACYJu4zH1dffbUefPBBjRo1StOmTdOGDRv0i1/8Qv/wD/8Q77caFJvNJpststiU+AAAwDxxj4/HH39cd999t2699VYdOXJEFRUVuuWWW3TPPffE+60GzW6zKWQYnO0CAICJ4h4feXl5WrZsmZYtWxbvb33G7DYpJGY+AAAwk6Xu7dJ7um2Y9gAAwDSWig97dNFpmPoAAMA0FouPSH1w1AUAAPNYMj5Y8wEAgHksFR+91/ogPgAAMI+l4sPOglMAAExnsfiIfDSY+QAAwDQWiw9mPgAAMJul4sPGglMAAExnqfhwRH9a4gMAAPNYKj64zgcAAOazZHww8wEAgHksFR9Hr/Nh7jgAALAyS8UHMx8AAJjPYvER+ch1PgAAMI/F4oPrfAAAYDZLxUfvmo8Q9QEAgGksFR+s+QAAwHyWjA/aAwAA81gqPo6eakt9AABgFkvFBwtOAQAwn6Xiw2FnzQcAAGazVHxwnQ8AAMxnqfiw9R52CZs8EAAALMxS8WFnwSkAAKazWHyw4BQAALNZMj64wikAAOaxVHxkOKLxwWEXAABMY6n46D3VNsSKUwAATGOp+MiIxkd3iJkPAADMYqn4cNgjPy5rPgAAMI+l4iMzuuajh/gAAMA0loqP2JqPEGs+AAAwi6Xio3fNBzMfAACYx1rx4Yj8uMQHAADmsVZ82LnIGAAAZrNUfDhip9qy5gMAALNYKj6Y+QAAwHzWig/WfAAAYDprxQczHwAAmM5S8cGaDwAAzGep+GDmAwAA81krPljzAQCA6SwVH0cvr058AABgFkvFR+9hl+4waz4AADCLpeLDwZoPAABMZ6n4yOxd88FhFwAATGOp+HBmRH7cQA+HXQAAMIul4sMVi4+QySMBAMC6LBUfvTMfQWY+AAAwjbXiI7rmI8gVTgEAMI2l4sOV6ZAkBbqJDwAAzGKp+GDmAwAA81kqPlyZLDgFAMBsloqP2MwHC04BADCNpeLDxdkuAACYLiHxcejQIX3jG99QUVGRsrKyNGPGDH366aeJeKtBcWVEF5wSHwAAmCYj3t+wqalJF154oS677DK98cYbKikp0c6dO1VQUBDvtxo0rvMBAID54h4fP//5z+Xz+fT000/Hnhs7dmy832ZIeg+79IQNhcJG7EZzAAAgeeJ+2OW1117TrFmz9Ld/+7cqLS3Vueeeq6eeeuqE2wcCAfn9/n6PROmd+ZCY/QAAwCxxj489e/boiSee0MSJE/XWW2/p29/+tm6//XY9++yzA26/dOlSeb3e2MPn88V7SDHEBwAA5rMZhhHX+8s7nU7NmjVLH3zwQey522+/XZ988onWrl173PaBQECBQCD2ud/vl8/nU0tLizweTzyHJsMwNP7HKxU2pI9//AWVetxx/f4AAFiV3++X1+s9rb/fcZ/5KC8v19SpU/s9d9ZZZ+nAgQMDbu9yueTxePo9EsVms8VmPzjjBQAAc8Q9Pi688EJt376933M7duzQ6NGj4/1WQ+KO3t+lq5urnAIAYIa4x8cdd9yhDz/8UA899JB27dql559/Xr/61a+0aNGieL/VkGRH46MjSHwAAGCGuMfHBRdcoJdfflkvvPCCpk+frgceeEDLli3TDTfcEO+3GpJsV+TsYuIDAABzxP06H5L0la98RV/5ylcS8a3PWI6zd+ajx+SRAABgTZa6t4skZTk57AIAgJksFx85zt7DLsx8AABgBsvFBzMfAACYy3LxcXTmg/gAAMAMlouPLBacAgBgKsvFR44rEh/tAWY+AAAwg+XiIzt62KWTwy4AAJjCgvERnfngsAsAAKawXHzkMPMBAICpLBcf2dE1H20BZj4AADCD5eIjz50pSWrtIj4AADCD5eLD444cdmnp7DZ5JAAAWJPl4sObFZn58HcRHwAAmMFy8eGJxkdboEfhsGHyaAAAsB7rxUd0zYdhsO4DAAAzWC4+nBl2ZWVGznjh0AsAAMlnufiQJE8Wi04BADCLNeMjeujFT3wAAJB0lowPzngBAMA8loyP3jNemjuIDwAAks2S8VGY45QkNbQHTR4JAADWY8n4KMqNxEcj8QEAQNJZMj6Kc1ySpIa2gMkjAQDAeiwZH70zHxx2AQAg+SwaH5GZj/o24gMAgGSzZnz0LjjlsAsAAElnyfgojs58NLYHubkcAABJZsn4KMiJXOejJ2xwoTEAAJLMkvHhynAozx25vwvrPgAASC5Lxod09NBLPes+AABIKsvGR5knEh+1/i6TRwIAgLVYNj7KvVmSpMMtxAcAAMlk2fgY4XVLkmqIDwAAksqy8VEejY/q5k6TRwIAgLVYOD4ih11qWPMBAEBSWTg+IjMfrPkAACC5LB8fda0BdXWHTB4NAADWYdn4KMxxKsfpkCQdYt0HAABJY9n4sNls8hVmS5IONHaYPBoAAKzDsvEhSaN646OB+AAAIFksHR+ji5j5AAAg2SwdH6M47AIAQNJZOj58HHYBACDpLB0f44pzJUl7G9oVChsmjwYAAGuwdHyMLMiSK8OuYE9YVRx6AQAgKSwdHw67TeNLIrMfu460mTwaAACswdLxIUkTyyLxsZP4AAAgKSwfHxNKeuOj1eSRAABgDZaPj96ZDw67AACQHJaPjwmleZKknbVtnPECAEASWD4+xhbnKCvToc7ukPbWM/sBAECiWT4+HHabplZ4JEmbDrWYPBoAAIY/y8eHJM0Y6ZUkbTroN3kkAAAMf8SHpOnR+PgLMx8AACQc8aGjMx+bq1sUZtEpAAAJRXxIGl+SI3emXe3BkPbUt5s9HAAAhrWEx8fDDz8sm82m733ve4l+qyHLcNg1vSIy+7HhQJPJowEAYHhLaHx88skn+uUvf6mzzz47kW8TFxeMLZQkfby30eSRAAAwvCUsPtra2nTDDTfoqaeeUkFBQaLeJm5m98bHPuIDAIBESlh8LFq0SFdddZXmz59/0u0CgYD8fn+/hxnOH10gu03a39ChWn+XKWMAAMAKEhIfK1as0Pr167V06dJTbrt06VJ5vd7Yw+fzJWJIp+RxZ8YuNsahFwAAEifu8VFVVaXvfve7eu655+R2u0+5/Z133qmWlpbYo6qqKt5DOm0XjGHdBwAAiRb3+Fi3bp2OHDmi8847TxkZGcrIyNC7776rxx57TBkZGQqFQv22d7lc8ng8/R5mmRNd9/HhngbTxgAAwHCXEe9v+IUvfEGbNm3q99xNN92kKVOm6Ec/+pEcDke83zJuPjeuSHabtPNImw42daiyINvsIQEAMOzEPT7y8vI0ffr0fs/l5OSoqKjouOdTTX62U+ePLtAn+5r0zrYjunHuGLOHBADAsMMVTo9x+ZQySdKfth0xeSQAAAxPcZ/5GMjq1auT8TZxcfmUUv38zW36YHeDOoMhZTlT9zARAADpiJmPY0wqy9XI/CwFesL6YHe92cMBAGDYIT6OYbPZdPmUUkkcegEAIBGIjwFcflYkPt7aXKueUNjk0QAAMLwQHwO4aEKxCrIzVd8W0Ae7ueYHAADxRHwMINNh11Vnl0uSXtl4yOTRAAAwvBAfJ/BX546UJL31lxp1BkOn2BoAAJwu4uMEzhtVIF9hltqDIb29tdbs4QAAMGwQHydgs9l07czI7McrGzj0AgBAvBAfJ3FNND7e21GnutaAyaMBAGB4ID5OYkJprmb68tUTNvTip1VmDwcAgGGB+DiFb3xutCTp+Y8OKBQ2TB4NAADpj/g4ha+cXa787Ewdau7UO1zxFACAM0Z8nII706HrZvkkSb9es9fk0QAAkP6Ij9OwcN4YZdhtWrunQZsOtpg9HAAA0hrxcRoq8rN09TkVkqRfvrfb5NEAAJDeiI/T9K2Lx0mSVm46rN11bSaPBgCA9EV8nKapFR7NP6tUYUP6tz/tMns4AACkLeJjEL43f5Ik6dWNh5j9AABgiIiPQZg+0qv5Z5UpbEi/eHuH2cMBACAtER+D9P0rJslmk17/82FtONBk9nAAAEg7xMcgnVXu0d+cVylJemjlVhkGVz0FAGAwiI8h+P4Vk+XOtOuTfU1auanG7OEAAJBWiI8hGOF16+ZLxkuSHvjvLWoP9Jg8IgAA0gfxMUS3fn68fIVZqvF36bFVO80eDgAAaYP4GCJ3pkP3XT1NUuSeL9tq/CaPCACA9EB8nIEvnFWmK6aWqSds6Ae/+7O6Q2GzhwQAQMojPs7Qz66dLm9WpjYdatEv3+W+LwAAnArxcYZKPW7d99WpkqT/tWqntlRz+AUAgJMhPuLg2pkj9cWpZeoOGbp9xQZ1BDn7BQCAEyE+4sBms+nhv56h0jyXdh1p00//zxazhwQAQMoiPuKkKNelZdfPlM0mrfikSq99Vm32kAAASEnERxzNG1+s2y6bIEla8l9/5vRbAAAGQHzE2e1fmKiLJhSrIxjSzb9dp+aOoNlDAgAgpRAfcZbhsOvxvztXvsIsHWjs0Hee36Aerv8BAEAM8ZEABTlOPfX3s5SV6dCaXfW6+9XN3P0WAIAo4iNBpozwxBagvvDxAS1/Z5fZQwIAICUQHwm0YNqI2P1fHv3DDv1+3UGTRwQAgPmIjwRbOG+Mbrl0nCTpR//1Z735lxqTRwQAgLmIjyT40YIp+uvzRioUNnTbC+u1amut2UMCAMA0xEcS2O02PfL/nK2vnF2u7pChb///6/XejjqzhwUAgCmIjyTJcNj1r9fN1IJpZQqGwvrWbz/Vn7YxAwIAsB7iI4kyHXY9/nfnaf5ZZQr0hHXzb9dxGXYAgOUQH0nmzLDriW+cp2tmVqgnbOi7Kzbo+Y8OmD0sAACShvgwQabDrn/92kzdMGeUDEP68cub9L/+uJMLkQEALIH4MIndbtPPrp2uWz8/XpL0r3/coe+/+JkCPSGTRwYAQGIRHyay2Wz64Zem6KG/miGH3aaXNhzSjb/+WE3t3IwOADB8ER8p4OtzRunpb16gPFeGPt7bqGv//X1tq/GbPSwAABKC+EgRl0wq0e+/PU8j87O0v6FD1y5/X69uPGT2sAAAiDviI4VMHpGn/3PbRbp4YrG6usP67oqNuu+1zQr2hM0eGgAAcUN8pJjCHKeeuWm2brt8giTpmQ/26bpfrdWBhg6TRwYAQHwQHynIYbfp+1dM1n/8/SzluTO04UCzvvzY/+il9Qc5HRcAkPaIjxQ2f2qZ3vjuxbpgTIHaAj1a/OJn+u6KjWrp7DZ7aAAADBnxkeIqC7K14ua5+v4XJ8lht+m1z6p15bL39C43pgMApCniIw047Dbd9oWJ+t0/z9WowmxVt3Rp4W8+1v/7u8/U0sEsCAAgvRAfaeS8UQV647sX65vzxshmk36/7qDm/+u7emtzjdlDAwDgtBEfaSbHlaH7vjpNv7tlrsaV5KiuNaBb/vc63fK/P9XBJs6IAQCkvrjHx9KlS3XBBRcoLy9PpaWluvbaa7V9+/Z4v43lzRpTqJW3X6xbPz9eDrtNb22u1fxfvKvHV+1UVzf3hwEApK64x8e7776rRYsW6cMPP9Tbb7+t7u5uXXHFFWpvb4/3W1meO9OhH35pilbefrFmjy1UV3dY/9/bO/SlZe/pnW1HzB4eAAADshkJvnBEXV2dSktL9e677+qSSy455fZ+v19er1ctLS3yeDyJHNqwYhiGXvusWg++vlVHWgOSpM9PLtGSK6doygj2IwAgsQbz9zsj0YNpaWmRJBUWFg74eiAQUCAQiH3u93NDtaGw2Wy6ZuZIXT6lVI//aZd+s2avVm+v03s76vQ351dq8Rcna4TXbfYwAQBI7MxHOBzWV7/6VTU3N2vNmjUDbnPffffp/vvvP+55Zj7OzL76dj3y1jat3BQ5E8adadc/XTROt1w6TnnuTJNHBwAYbgYz85HQ+Pj2t7+tN954Q2vWrFFlZeWA2ww08+Hz+YiPOFl/oEkPvb5Vn+5vkiTlZ2fqWxeP08J5Y5TrSvjEFwDAIlIiPr7zne/o1Vdf1XvvvaexY8ee9tex5iP+DMPQH7bU6udvbtOeusjC34LsTN18yXj9/dzRyiFCAABnyNT4MAxDt912m15++WWtXr1aEydOHNTXEx+JEwobeu2zQ3ps1S7trY9ESGGOUzdfMk43fo4IAQAMnanxceutt+r555/Xq6++qsmTJ8ee93q9ysrKOuXXEx+J1xMK67XPqvXYqp3a1xC5MJk3K1N/P3e0Fs4bo+Jcl8kjBACkG1Pjw2azDfj8008/rW9+85un/HriI3l6QmG9srFay985OhPiyrDrb86v1LcuHqcxxTkmjxAAkC5SYs3HUBEfyRcKG3p7S62efHe3NlY1S5LsNunK6eX6x4vH6lxf/gmjEgAAifjAEBmGoY/3NuqX7+3Rn/pcIfXsSq8Wzh2jq84ulzvTYeIIAQCpivjAGdtW49ev/2evXv2sWsGesCSpKMep62f79I3PjVa599TrdwAA1kF8IG4a24N64eMDeu7D/apu6ZIkOew2XTG1TH83e5QumlAsu51DMgBgdcQH4q4nFNbbW2r1zAf79NHextjzI/OzdN0FPv3trEpmQwDAwogPJNS2Gr9e+OiAXt5wSP6uHkmRBaqXTirR9bNH6fIppcp0xP2GyQCAFEZ8ICm6ukN68y81euHjA/1mQ4pznbr6nAr99bmVmj7Sw5kyAGABxAeSbk9dm/7z0yr917qDqm8Lxp6fUJqrvzp3pK6ZWaHKgmwTRwgASCTiA6bpDoX1Pzvr9NL6Q3p7S60C0TNlJGnO2EL91bkj9aXpI5Sf7TRxlACAeCM+kBJau7r1xl9q9MqGQ1q7p0G9v2kZdpsunFCsq84u14KpI+TNzjR3oACAM0Z8IOVUN3fqtc+q9cqGQ9pW0xp7PtMRDZEZ5bqCEAGAtEV8IKXtrmvTyj8f1uubDg8YIgumjdAXzipVaZ7bxFECAAaD+EDa2HWkTSs3HdbKY0LEZpPO9eXri1NH6ItTyzShNNfEUQIAToX4QFradaRVb22u1R821+izgy39XhtXkqMvTi3TFVPLNNNXIAdXVQWAlEJ8IO3VtHTp7a21entLrdburld36OivaUF2pi6ZVKLLJpfqkkklKszhzBkAMBvxgWGltatbq7fX6e0ttXpn+xG1Rq+qKkUOz5xTma/LJpfqsiklml7h5V4zAGAC4gPDVk8orPUHmvXO9iNavb1OWw/7+71enOvUJRNLdNHEYl04oVhlHhatAkAyEB+wjJqWLr2744je2VanNbvq1Rbo6ff6xNJcXTSxWBdNKNaccUXKdWWYNFIAGN6ID1hSsCesT/c3as3Oeq3ZVa9Nh1rU97c7w27TuaPydeGESIycXZkvZwY3wAOAeCA+AEnNHUGt3d2g/9lVr/d31Wt/Q0e/192Zds0aXajPjSvUnHFFOocYAYAhIz6AAVQ1dmjNrnqt2VmvtXsa1Nge7Pe6O9Ou80cXaM7YIn1uXJHO8XnlynCYNFoASC/EB3AKhmFo55E2fbinQR/tadSHexrUcEyMuDLsmunL16wxBZo1plDnjSqQN4vLvwPAQIgPYJAMw9CuaIx8uLdRH+1pUH1b/xix2aRJpXnRGCnQrNGFqizIks3Gqb0AQHwAZ8gwDO2ua9On+5r0yb4mrdvfqH3HrBmRpDKPS7NGF+rcUfk6d1SBplV45M7kUA0A6yE+gASoaw1o3f7GSJDsb9LmQy3qCff/n0+mw6ap5R6dO6pA547K10xfvkYVZjM7AmDYIz6AJOgMhrSxqlnr9jdqY1WzNhxoPm7diCQV5jh1ri8SIuf48jVjpFcFXBIewDBDfAAmMAxDB5s6tf5AkzYcaNbGqmZtqfYrGAoft62vMEtnj8zX2ZVezaj0avpIrzxuFrMCSF/EB5AiAj0hban2a8OBZm2oatZfDrVob337gNuOK87RjEqvZoyMPKZWeJRHkABIE8QHkMJaOrr1l+oW/flgizYdatafD7boYFPngNuOKcrW1AqPplV4ox89Ks3jfjUAUg/xAaSZxvagNh1q0aaDzfrsYIs2H2pRdUvXgNuW5Lk0LRoi0yq8Oqvco1GF2XJwN18AJiI+gGGgsT2oLdV+ba5u0eboxz317Rrof7HuTLsmleVpclmeJo84+ijJdXGmDYCkID6AYaoj2KOth1u1JRYkfu2obVWg5/hFrVLkTJtjg2RyWZ5yuLsvgDgjPgALCYUN7Wto1/aa1qOP2lbtaxh4lkSKnG0zucyjKX2iZGxxjjId3FgPwNAQHwDUGQxp55FWbTsmSupaAwNu73TYNa4kJxokHk0qy9X4klxVFmQpgygBcArEB4ATamgLaHvt0SDZVtOqHbWt6giGBtze6bBrTHG2JpRGYqT3Ma4kh8M3AGKIDwCDEg4bOtTcGZ0l8WtbTat2HWnT3vr2E64nkaQKr1vjY1GSo/EluZpQmquSPBa6AlZDfACIi1DYUHVzp3bVtWn3kTbtrmvX7ui/B7qUfK88V4bGlfYPkvEluRpdlM26EmCYIj4AJFxTe1B76tu0+0g0SOratOtImw40dih8gv+qZNhtGlWUrbFFORpdlKMxxdkaU5SjMUU5qsh3s7YESGOD+fvNAVsAQ1KQ49T5OYU6f3Rhv+cDPSHtb+iIzpREZkt2Rf/dEQxpT1279tQdf4n5TIdNvoJsjS7K1pjiSJCMLsrW2OIcjcxn0SswnDDzASApDMNQjb9Le+rata+hXfvq27WvoUP7GyIfgydZW5Jht8lXGA2TohyNKcrW6GigVBZkcSgHSAEcdgGQVsLhSJj0DZK99e3a39ChfQ0nX/TqsNtUWZClMUU5GlWYLV9hlnwF2fIVRh7eLG7OByQDh10ApBW73aaK/CxV5Gdp3oT+r4XDhmpbu7SvPhIi+xratb/Pv7u6w9rf0KH9DR0Dfm+POyMSIgXZGlWULV9Bliqjn1cWZMmd6UjCTwigL2Y+AKQtwzB0pDWgvfWRwzhVTR2qauzUgcYOHWzqUH3bic/I6VXmcR2dKYmGyajorMkIj5sb9gGnicMuAKDIvXCqGjtV1dhxXJhUNXao/QQXVuuV6YjMyFQWZGlkfpZG5merIt+tkQVZqszP1givW84M1psAEoddAECSlO3MiN275liGYaipo1sHGjv6xUnvvw81dao7ZJz0kI7NJpXmuTQyesgoEiWRjxX5kWDJc7PmBDgW8QHAkmw2mwpznCrMcWqmL/+410PRRbAHGjp0qLlT1c2dOtTUefTfzZ0K9IRV6w+o1h/Q+gPNA76Px53Rb/akok+clHvdKs3j0A6sh/gAgAE47LbooZasAV83DEP1bcFYiPSGSd9/t3R2y9/VI3/0Hjonep/SPJdGeN0q97o1whOJkt7Py/OzVJrn4nRiDCvEBwAMgc1mU0meSyV5Lp0zwMyJJLUFeo6Lk94ZlMMtXarxdykUNnS4pUuHW7q04YTvJZXkuvpESdbROPFGZ1A8LrkyOHMH6YH4AIAEyXVlaFJZniaVHb/mRIoc2mloC0TjIxok0RCpaenSYX+nalq61B2KnNVzpDWgzw62nPD9inOdGtFn9qQ8//jZFE4tRiogPgDAJA67TaUet0o97hPOnoTDhho7gqpp6VJ1c6dq/H3ipKUzFiuBnrDq24KqbwvqL4f8J3xPb1amyjwulXncKsmLfCyLfiz1uFXmiczmMIuCRCI+ACCF2e02Fee6VJzr0vSR3gG3MQxDzR3dA8+g+COfH27uUmd3SC2d3Wrp7NaO2raTvm9BdubRIMlzqTQaLKV57n7xwloUDAXxAQBpzmazqSDHqYIcp6ZWDHx9BcMw5O/s0ZHWrugZOl060tr7sc9z/oCCobCaOrrV1NF9woWyvYpynLEZk7JomJR43CrJjcyglOZFwinLyUwKjiI+AMACbDabvNmZ8mZnauIJ1qBIkUhp6eyOxUi/SPEHVNsa+XikNbIWpaE9qIb2oLYePvn757kyVJznikVJ7JHrUnGeUyW5kZmUolwnsykWQHwAAGJsNpvys53Kz3YOeHG2XuGwoaaOoGqjIXKkN1aisyj1bQHVtUYegZ6wWgM9ag30aG99+ynHUJjj7BMl/WOlOPdotBRkO2XnGilpifgAAAya3W5TUa5LRbkuTdWJL6VtGIZaAz2xEOkbJXWtAdW1HX2+vi2oUNhQY3tQje1Bba89+RgcdpuKc52xKCnKcak416miXKeKclwqzHWqOCcym1KY4+RMnxRCfAAAEsZms8njzpTHnanxJbkn3bZ3NqXumECJBUuf55s6uhUKG7ErzJ6OPFdGJExyXSrKiXwsznXG/l2U64xGTGTmhyvPJg7xAQBICX1nU6aMOPm2wZ6wGtuDqmuNHPZpaAuqvj2ghragGtoCamiPnHbcGH2uJ2zEDv3sO8G9evqNxSYVZB+dRekbJkdD5ehrua4M2WzEyulKWHwsX75c//Iv/6Kamhqdc845evzxxzV79uxEvR0AwEKcGfbIBdW8bkkDn4Lcq/dMn75xUt8ejZS2oBraI4d8eqOluaNbYUOxxbTSyU9L7h1PcY5ThblOFea4VJidqYKcyKxKQY5ThdnRjzlOFWQ7VZCdqQwLL6xNSHz853/+pxYvXqwnn3xSc+bM0bJly7RgwQJt375dpaWliXhLAAAG1PdMn/Elp96+OxRWU0cwGirHxEnfz6Mx0xEMKdgTVnVLl6pbuk57XB53hopyXSrIzoxFSeEAsVIY/TzPnTFsFtjaDMMw4v1N58yZowsuuED/9m//JkkKh8Py+Xy67bbbtGTJkpN+rd/vl9frVUtLizyeEy9iAgAgFXQGQ7EQqW8LqLE9qKaOoBrbu9XUHlRjRzD2sbE9qJbObg3lL6/DblNBdmZk5iQaJIW5fUMl82jARD9mOx1JOxw0mL/fcZ/5CAaDWrdune68887Yc3a7XfPnz9fatWuP2z4QCCgQOLpYyO8/8WWBAQBINVlOhyqd2aosyD6t7XtCYbV0dscC5WisRCPlmGBpau9WW6BHobARu4T+6XJl2FWQ7VR+LFoylZ/tVGVBlm79/ISh/shnLO7xUV9fr1AopLKysn7Pl5WVadu2bcdtv3TpUt1///3xHgYAACkpw2GPLaw9XYGekJo7umOnIR8XLB3dsXBp6oisVQn2hBXoCavGH7mDcl/jSnKGV3wM1p133qnFixfHPvf7/fL5fCaOCACA1OLKcKjM41CZx31a2xuGoY5gKBYjTR3dao7OpjR1dCvPbe6f/7i/e3FxsRwOh2pr+18dpra2ViNGHH/ulMvlkst1+vUHAABOzmazKceVoRxXhnyFp3c4KJnifp6P0+nU+eefr1WrVsWeC4fDWrVqlebOnRvvtwMAAGkmIfMuixcv1sKFCzVr1izNnj1by5YtU3t7u2666aZEvB0AAEgjCYmP6667TnV1dbrnnntUU1OjmTNn6s033zxuESoAALCehFzn40xwnQ8AANLPYP5+W/fargAAwBTEBwAASCriAwAAJBXxAQAAkor4AAAASUV8AACApCI+AABAUhEfAAAgqYgPAACQVObeU3cAvRdc9fv9Jo8EAACcrt6/26dz4fSUi4/W1lZJks/nM3kkAABgsFpbW+X1ek+6Tcrd2yUcDqu6ulp5eXmy2Wxx/d5+v18+n09VVVXcNyaB2M/JwX5ODvZz8rCvkyNR+9kwDLW2tqqiokJ2+8lXdaTczIfdbldlZWVC38Pj8fCLnQTs5+RgPycH+zl52NfJkYj9fKoZj14sOAUAAElFfAAAgKSyVHy4XC7de++9crlcZg9lWGM/Jwf7OTnYz8nDvk6OVNjPKbfgFAAADG+WmvkAAADmIz4AAEBSER8AACCpiA8AAJBUwy4+li9frjFjxsjtdmvOnDn6+OOPT7r97373O02ZMkVut1szZszQypUrkzTS9DaY/fzUU0/p4osvVkFBgQoKCjR//vxT/t8FEYP9fe61YsUK2Ww2XXvttYkd4DAx2P3c3NysRYsWqby8XC6XS5MmTeK/HadpsPt62bJlmjx5srKysuTz+XTHHXeoq6srSaNNP++9956uvvpqVVRUyGaz6ZVXXjnl16xevVrnnXeeXC6XJkyYoGeeeSbh45QxjKxYscJwOp3Gb37zG2Pz5s3Gt771LSM/P9+ora0dcPv333/fcDgcxiOPPGJs2bLF+MlPfmJkZmYamzZtSvLI08tg9/PXv/51Y/ny5caGDRuMrVu3Gt/85jcNr9drHDx4MMkjTy+D3c+99u7da4wcOdK4+OKLjWuuuSY5g01jg93PgUDAmDVrlvHlL3/ZWLNmjbF3715j9erVxsaNG5M88vQz2H393HPPGS6Xy3juueeMvXv3Gm+99ZZRXl5u3HHHHUkeefpYuXKlcddddxkvvfSSIcl4+eWXT7r9nj17jOzsbGPx4sXGli1bjMcff9xwOBzGm2++mdBxDqv4mD17trFo0aLY56FQyKioqDCWLl064PZf+9rXjKuuuqrfc3PmzDFuueWWhI4z3Q12Px+rp6fHyMvLM5599tlEDXFYGMp+7unpMebNm2f8x3/8h7Fw4ULi4zQMdj8/8cQTxrhx44xgMJisIQ4bg93XixYtMi6//PJ+zy1evNi48MILEzrO4eJ04uOHP/yhMW3atH7PXXfddcaCBQsSODLDGDaHXYLBoNatW6f58+fHnrPb7Zo/f77Wrl074NesXbu23/aStGDBghNuj6Ht52N1dHSou7tbhYWFiRpm2hvqfv7pT3+q0tJS/eM//mMyhpn2hrKfX3vtNc2dO1eLFi1SWVmZpk+froceekihUChZw05LQ9nX8+bN07p162KHZvbs2aOVK1fqy1/+clLGbAVm/R1MuRvLDVV9fb1CoZDKysr6PV9WVqZt27YN+DU1NTUDbl9TU5Owcaa7oeznY/3oRz9SRUXFcb/wOGoo+3nNmjX69a9/rY0bNyZhhMPDUPbznj179Kc//Uk33HCDVq5cqV27dunWW29Vd3e37r333mQMOy0NZV9//etfV319vS666CIZhqGenh798z//s3784x8nY8iWcKK/g36/X52dncrKykrI+w6bmQ+kh4cfflgrVqzQyy+/LLfbbfZwho3W1lbdeOONeuqpp1RcXGz2cIa1cDis0tJS/epXv9L555+v6667TnfddZeefPJJs4c27KxevVoPPfSQ/v3f/13r16/XSy+9pNdff10PPPCA2UPDGRo2Mx/FxcVyOByqra3t93xtba1GjBgx4NeMGDFiUNtjaPu516OPPqqHH35Yf/zjH3X22Wcncphpb7D7effu3dq3b5+uvvrq2HPhcFiSlJGRoe3bt2v8+PGJHXQaGsrvc3l5uTIzM+VwOGLPnXXWWaqpqVEwGJTT6UzomNPVUPb13XffrRtvvFH/9E//JEmaMWOG2tvbdfPNN+uuu+6S3c7//3ymTvR30OPxJGzWQxpGMx9Op1Pnn3++Vq1aFXsuHA5r1apVmjt37oBfM3fu3H7bS9Lbb799wu0xtP0sSY888ogeeOABvfnmm5o1a1YyhprWBrufp0yZok2bNmnjxo2xx1e/+lVddtll2rhxo3w+XzKHnzaG8vt84YUXateuXbG4k6QdO3aovLyc8DiJoezrjo6O4wKjN/oMbksWF6b9HUzoctYkW7FiheFyuYxnnnnG2LJli3HzzTcb+fn5Rk1NjWEYhnHjjTcaS5YsiW3//vvvGxkZGcajjz5qbN261bj33ns51fY0DHY/P/zww4bT6TR+//vfG4cPH449WltbzfoR0sJg9/OxONvl9Ax2Px84cMDIy8szvvOd7xjbt283/vu//9soLS01fvazn5n1I6SNwe7re++918jLyzNeeOEFY8+ePcYf/vAHY/z48cbXvvY1s36ElNfa2mps2LDB2LBhgyHJ+MUvfmFs2LDB2L9/v2EYhrFkyRLjxhtvjG3fe6rtD37wA2Pr1q3G8uXLOdV2KB5//HFj1KhRhtPpNGbPnm18+OGHsdcuvfRSY+HChf22f/HFF41JkyYZTqfTmDZtmvH6668necTpaTD7efTo0Yak4x733ntv8geeZgb7+9wX8XH6BrufP/jgA2POnDmGy+Uyxo0bZzz44INGT09Pkkedngazr7u7u4377rvPGD9+vOF2uw2fz2fceuutRlNTU/IHnibeeeedAf9727tfFy5caFx66aXHfc3MmTMNp9NpjBs3znj66acTPk6bYTB3BQAAkmfYrPkAAADpgfgAAABJRXwAAICkIj4AAEBSER8AACCpiA8AAJBUxAcAAEgq4gMAACQV8QEAAJKK+AAAAElFfAAAgKQiPgAAQFL9Xz3HZgRXV/DjAAAAAElFTkSuQmCC\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Define negative log likelihood function\n",
    "nll = lambda pred, actual: -(actual * np.log(pred) + (1-actual) * np.log((1-pred)))\n",
    "\n",
    "# Plot negative log likelihood against our simulated output probabilities\n",
    "plt.plot(output_probs, nll(output_probs, 1))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can see that negative log likelihood gets very large as our prediction gets closer to `0`.  This loss function will penalize big errors in predictions more than mean squared error.\n",
    "\n",
    "## Training Loop - Binary Classification\n",
    "\n",
    "We now know enough to try binary classification using our neural network.  We'll run the network normally, but apply the sigmoid function to the output layer, then measure error with negative log likelihood.\n",
    "\n",
    "In the last lesson, we trained a neural network to make predictions.  We'll use the same code as last time, but this time we'll define our neural network using a class.  This helps us avoid rewriting all of the same code from last time again.\n",
    "\n",
    "We'll define a neural network with 2 layers.  Layer 1 will take in 13 input features, and convert them into 25 features.  Layer 2 will take those 25 features, and turn them into a single output."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "outputs": [],
   "source": [
    "# Append to the path, so we can import our neural network\n",
    "sys.path.append(os.path.abspath(\"../nnets\"))\n",
    "from dense import Dense\n",
    "\n",
    "class ClassificationNet():\n",
    "    def __init__(self, output_size=1):\n",
    "        # Setup 2 neural network layers.  Each Dense class is a single network layer.\n",
    "        # We don't use relu activation on the second layer, but we do in the first layer.\n",
    "        self.layer1 = Dense(input_size=13, output_size=25)\n",
    "        self.layer2 = Dense(input_size=25, output_size=output_size, activation=False)\n",
    "        self.last_input = None\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        # In the forward pass, we take in input data, and run our 2 layers over the data.\n",
    "        x = self.layer1.forward(x)\n",
    "        x = self.layer2.forward(x)\n",
    "        return x\n",
    "\n",
    "    def backward(self, grad, lr):\n",
    "        # In the backward pass, we take the gradient and learning rate, and use them to adjust parameters in each layer.\n",
    "        grad = self.layer2.backward(grad, lr)\n",
    "        grad = self.layer1.backward(grad, lr)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The forward and backward pass will mostly work the same way they did in the last lesson.  The main difference is in how we calculate the gradient of the loss function and backpropagate it across the sigmoid activation.\n",
    "\n",
    "As mentioned earlier, our network will output a single number in the last layer - let's call this $z$.  We'll then run the sigmoid function over $z$ to get the output probability $p$.  We'll then use a loss function $L$ (negative log likelihood) to calculate how far we were from the actual target.\n",
    "\n",
    "The derivative of the loss function $L$ with respect to the predicted probabilities $p$ is $\\frac{\\partial L}{\\partial p}=\\frac{p-y}{p(1-p)}$.  And the derivative across the sigmoid function of the probabilities $p$ with respect to the neural network output $z$ is $\\frac{\\partial p}{\\partial z}=p(1-p)$.  If we want to compute the partial derivative of the loss $L$ with respect to the neural network output $z$, we multiply the two by the chain rule:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial z} = \\frac{\\partial L}{\\partial p} * \\frac{\\partial p}{\\partial z}$$\n",
    "$$\\frac{\\partial L}{\\partial z} = \\frac{p-y}{p(1-p)} * \\frac{\\partial p}{\\partial z}=p(1-p)$$\n",
    "$$\\frac{\\partial L}{\\partial z} = p - y$$\n",
    "\n",
    "Luckily for us, the denominator multiplies out, and we're left with $p-y$, or the predicted probabilities minus the target.  So in one step, we'll calculate the gradient with respect to the output of our neural network:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "outputs": [],
   "source": [
    "# Computes the gradient with respect to the outputs of the neural network\n",
    "# Input will be the sigmoid function applied to the neural network outputs\n",
    "nll_grad = lambda pred, actual: pred - actual"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can see that this gives us a clear gradient that we can follow with gradient descent:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "outputs": [
    {
     "data": {
      "text/plain": "[<matplotlib.lines.Line2D at 0x1775a0790>]"
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABBaklEQVR4nO3dd3xUBb7+8c9MyqSQQiCFQCihhZoCiqAsKiiIDUVaWBUvirsSQEFdsGFbcRUbZXXr1b2XUAVEZXERZFcRQUlCTUKHUJIQQjpJJjPn98fu5ndxAQlkcjKT5/16zR85OWfOk0My5+F8p1gMwzAQERERcRNWswOIiIiI1IXKi4iIiLgVlRcRERFxKyovIiIi4lZUXkRERMStqLyIiIiIW1F5EREREbei8iIiIiJuxdvsAPXN6XRy8uRJgoKCsFgsZscRERGRy2AYBqWlpURHR2O1XvraiseVl5MnTxITE2N2DBEREbkCOTk5tGnT5pLreFx5CQoKAv75wwcHB5ucRkRERC5HSUkJMTExtefxS/G48vLvUVFwcLDKi4iIiJu5nKd86Am7IiIi4lZUXkRERMStqLyIiIiIW1F5EREREbei8iIiIiJuReVFRERE3IrKi4iIiLgVlRcRERFxKyovIiIi4lYapLwsXLiQ9u3b4+fnR79+/di2bdsl11++fDlxcXH4+fnRq1cv1q5d2xAxRURExA24vLwsXbqU6dOnM3v2bNLS0oiPj2fo0KHk5+dfcP1vv/2WcePGMXHiRNLT0xkxYgQjRoxg9+7dro4qIiIibsBiGIbhyh3069ePa665hgULFgDgdDqJiYlhypQpzJw58z/WHzNmDOXl5Xz22We1y6677joSEhL44IMPfnJ/JSUlhISEUFxcrM82EhERcRN1OX+79MpLdXU127dvZ8iQIf9/h1YrQ4YMYcuWLRfcZsuWLeetDzB06NCLrl9VVUVJScl5NxEREal/lXYHs1buZPkPOabmcGl5KSgowOFwEBkZed7yyMhIcnNzL7hNbm5undafM2cOISEhtbeYmJj6CS8iIiK1DuSXMWLhZhZvy+HFNXsoqqg2LYvbv9po1qxZFBcX195ycsxtgyIiIp7m4+3HuXP+N2TlltKymY3f3d+X0ABf0/J4u/LOW7ZsiZeXF3l5eectz8vLIyoq6oLbREVF1Wl9m82GzWarn8AiIiJSq6K6hhc+2cOK7ccBGNCxBe+OTSAiyM/UXC698uLr60ufPn3YsGFD7TKn08mGDRvo37//Bbfp37//eesDrF+//qLri4iISP3bl1fK3Qs2s2L7cawWmH5LF/5nYj/Tiwu4+MoLwPTp03nwwQfp27cv1157Le+++y7l5eU89NBDADzwwAO0bt2aOXPmADBt2jQGDRrEW2+9xe23386SJUv44Ycf+P3vf+/qqCIiIk2eYRgs+yGH2Wv2UGl3EhFk472xifTv2MLsaLVcXl7GjBnD6dOneeGFF8jNzSUhIYF169bVPin32LFjWK3//wLQgAEDSE1N5bnnnuOZZ56hc+fOrF69mp49e7o6qoiISJNWVlXDc6t2sTrjJAADO7fknTEJtGzWuJ6e4fL3eWloep8XERGRutt7soSU1DQOFZTjZbUw49Yu/OJnHbFaLQ2y/7qcv11+5UVEREQaL8MwSN12jJc+3Ut1jZNWIX7MG5fINe3DzI52USovIiIiTVRppZ2ZK3fx+c5TANwcF8HcUfGEBZr3MujLofIiIiLSBO0+Uczk1DSOnqnA22rh6WFdefiG2AYbE10NlRcREZEmxDAM/rLlKL/+PJNqh5PWof7MT04kqW1zs6NdNpUXERGRJqL4nJ1frdjJuj3//MidW7tH8uZ98YQE+JicrG5UXkRERJqAjJwiUlLTOH72HD5eFp4Z3o0JA9pjsTT+MdGPqbyIiIh4MMMw+NM3h/nNuizsDoO2YQEsSE6kd5tQs6NdMZUXERERD1VUUc2Ty3fwZWY+AMN7RfH6yN4E+7nXmOjHVF5EREQ80PajhUxJTedkcSW+3laev6M7P+/X1i3HRD+m8iIiIuJBnE6D3399iDe/yMbhNOjQMpAFyYn0iA4xO1q9UXkRERHxEGfKqpixfAebsk8DcFd8NK/d24tmNs863XvWTyMiItJEbT10hqlL0skrqcLmbeXFu3ow9poYjxgT/ZjKi4iIiBtzOg1+u+kAb6/fh9OAjuGBLByfRFyU5344scqLiIiImzpdWsX0ZRl8vb8AgHuTWvPK3T0J9LAx0Y959k8nIiLiob49UMC0pRmcLq3C38eLl+/uwai+MWbHahAqLyIiIm7E4TSYt2E/8zbuxzCgS2QzFiYn0TkyyOxoDUblRURExE3klVQybUk63x0qBGBM3xhevKsH/r5eJidrWCovIiIibuAf+07zxNIMzpRXE+DrxWv39GJEYmuzY5lC5UVERKQRq3E4eefLffx200EMA7q1CmZhciKx4c3MjmYalRcREZFG6lTxOaYuTuf7I2cBGN+vLc/f0R0/n6Y1JvoxlRcREZFG6KusfKYvy+BshZ1mNm9eH9mLO3pHmx2rUVB5ERERaUTsDidzv8jmd/84BEDP1sEsGJdE+5aBJidrPFReREREGonjZyuYsjid9GNFAEwY0J5Zw+OweTftMdGPqbyIiIg0An/bk8tTK3ZSfM5OkJ83b97Xm2E9W5kdq1FSeRERETFRdY2T1/+axZ83HwYgPiaUBeMSiQkLMDlZ46XyIiIiYpJjZypIWZzGzuPFADx8QweeHhaHr7fV5GSNm8qLiIiICf666xRPr9hJaVUNIf4+vDUqniHdI82O5RZUXkRERBpQpd3Ba2sz+cuWowD0adeceeMSaR3qb3Iy96HyIiIi0kAOF5STkprGnpMlAPxiUEdm3NoFHy+NiepC5UVERKQBrNlxkmdW7qKsqoawQF/eGh3PTV0jzI7lllReREREXKjS7uClT/eyeNsxAK5tH8a8cYlEhfiZnMx9qbyIiIi4yIH8MlJS08jKLcVigZSbOjFtcGe8NSa6KiovIiIiLrAy7TjPrd5NRbWDls18eWdMAgM7h5sdyyOovIiIiNSjiuoaZn+yh+XbjwMwoGML3h2TQESwxkT1ReVFRESknuzLK2XyojT255dhtcC0wV1IubkTXlaL2dE8isqLiIjIVTIMg+Xbj/PCJ7uptDuJCLLx3thE+ndsYXY0j6TyIiIichXKq2p4bvVuVqWfAGBg55a8MyaBls1sJifzXCovIiIiVyjzVAmTF6VxqKAcL6uF6bd04ZeDOmLVmMilVF5ERETqyDAMUrcd46VP91Jd4yQq2I/5yYlc0z7M7GhNgsqLiIhIHZRW2pm1chef7TwFwE1dw3lrdAJhgb4mJ2s6VF5EREQu0+4TxaSkpnHkTAXeVgtPD+vKwzfEakzUwFReREREfoJhGPxly1F+/Xkm1Q4nrUP9mTcukT7tmpsdrUlSeREREbmE4nN2Zn68k7/uzgXglu6RvHlfb0IDNCYyi8qLiIjIRezIKSJlcRo5hefw8bIw67ZuPHR9eywWjYnMpPIiIiLyI4Zh8OfNR3j9r5nYHQYxYf4sGJdEfEyo2dEElRcREZHzFFVU8+TynXyZmQfAbT2jeH1kb0L8fUxOJv+m8iIiIvIv24+eZUpqGieLK/H1svL8Hd34+XXtNCZqZFReRESkyXM6DX7/9SHe/CIbh9OgfYsAFiQn0bN1iNnR5AJUXkREpEkrLK9m+rIMNmWfBuDO+Gheu6cnQX4aEzVWKi8iItJkbTtcyNTF6eSWVGLztvLiXT0Ye02MxkSNnMqLiIg0OU6nwW83HeDt9ftwGhAbHsjC5CS6tQo2O5pcBpUXERFpUk6XVjF9WQZf7y8A4N7E1rwyoieBNp0S3YX+pUREpMn49mAB05ZkcLq0Cj8fKy/f3ZNRfdpoTORmVF5ERMTjOZwG8zfuZ96G/TgN6BLZjIXJSXSODDI7mlwBlRcREfFo+SWVTFuSwZZDZwAY3bcNL93VE39fL5OTyZVSeREREY/19f7TPLE0g4KyagJ8vfj1PT25J7GN2bHkKlldeeeFhYWMHz+e4OBgQkNDmThxImVlZZdcf8qUKXTt2hV/f3/atm3L1KlTKS4udmVMERHxMDUOJ3O/yOaBP2+joKyauKggPp1yg4qLh3DplZfx48dz6tQp1q9fj91u56GHHmLSpEmkpqZecP2TJ09y8uRJ5s6dS/fu3Tl69Ci/+MUvOHnyJCtWrHBlVBER8RCnis8xbXEG244UApDcry0v3NEdPx+NiTyFxTAMwxV3nJmZSffu3fn+++/p27cvAOvWrWP48OEcP36c6Ojoy7qf5cuX8/Of/5zy8nK8vX+6a5WUlBASEkJxcTHBwXq9vohIU/JVVj7Tl2VwtsJOM5s3c+7txZ3xl3e+EXPV5fztsrHRli1bCA0NrS0uAEOGDMFqtbJ169bLvp9//xCXU1xERKRpsjuczFmbyUMffs/ZCjs9Wwfz2ZQbVFw8lMsaQW5uLhEREefvzNubsLAwcnNzL+s+CgoKeOWVV5g0adJF16mqqqKqqqr265KSkisLLCIibulE0TmmpKaRdqwIgAf7t+OZ27th89aYyFPV+crLzJkzsVgsl7xlZWVddbCSkhJuv/12unfvzosvvnjR9ebMmUNISEjtLSYm5qr3LSIi7mH93jyGv/c1aceKCPLz5v3xSbx0d08VFw9X5ysvM2bMYMKECZdcJzY2lqioKPLz889bXlNTQ2FhIVFRUZfcvrS0lGHDhhEUFMSqVavw8bn4J3vOmjWL6dOn135dUlKiAiMi4uGqa5z8Zl0Wf/rmMADxbUJYkJxETFiAycmkIdS5vISHhxMeHv6T6/Xv35+ioiK2b99Onz59ANi4cSNOp5N+/fpddLuSkhKGDh2KzWZjzZo1+Pn5XXI/NpsNm81Wtx9CRETcVk5hBSmpaew4/s+30Zh4Qwd+NSwOX2+XvvuHNCIu+5fu1q0bw4YN45FHHmHbtm1s3ryZlJQUxo4dW/tKoxMnThAXF8e2bduAfxaXW2+9lfLycv70pz9RUlJCbm4uubm5OBwOV0UVERE3sW73KYbP+5odx4sJ8ffhDw/05fk7uqu4NDEufQnPokWLSElJYfDgwVitVkaOHMm8efNqv2+328nOzqaiogKAtLS02lciderU6bz7Onz4MO3bt3dlXBERaaQq7Q7mrM3koy1HAUhqG8r85CRah/qbnEzM4LL3eTGL3udFRMSzHCkoZ3JqGntO/vPVpI8OiuXJW7vi46WrLZ6kLudvvXmKiIg0Wp/uOMmslbsoq6qheYAPb49O4Ka4iJ/eUDyayouIiDQ6lXYHL3+2l9StxwC4tn0Y741LoFWIxkSi8iIiIo3MwdNlTF6URlZuKRYLTL6xE48P6Yy3xkTyLyovIiLSaKxKP86zq3ZTUe2gZTNf3hmTwMDOP/32HNK0qLyIiIjpzlU7mL1mN8t+OA5A/9gWvDc2gYjgS7/XlzRNKi8iImKq/XmlPLYojf35ZVgsMG1wZ6bc3Bkvq8XsaNJIqbyIiIgpDMNg+fbjvPDJbirtTsKDbLw3NoEBHVuaHU0aOZUXERFpcOVVNTy/ejcr008AMLBzS94Zk0DLZvq4F/lpKi8iItKgMk+VkJKaxsHT5VgtMOPWrvxyUEesGhPJZVJ5ERGRBmEYBou35fDSp3uoqnESFezHvHGJXNshzOxo4mZUXkRExOVKK+08s2o3n+44CcCNXcN5e3QCYYG+JicTd6TyIiIiLrX7RDEpqWkcOVOBl9XC00O78sjAWI2J5IqpvIiIiEsYhsH/fHeUVz/LpNrhpHWoP/PGJdKnXXOzo4mbU3kREZF6V3zOzqyVO1m7KxeAId0imTuqN6EBGhPJ1VN5ERGRerUjp4iUxWnkFJ7Dx8vCzNu68V/Xt8di0ZhI6ofKi4iI1AvDMPjz5iO8/tdM7A6DmDB/FoxLIj4m1Oxo4mFUXkRE5KoVVVTz1IqdrN+bB8BtPaN4fWRvQvx9TE4mnkjlRURErkrasbNMSU3nRNE5fL2sPHdHN+6/rp3GROIyKi8iInJFnE6DP3x9iDe/yKbGadCuRQALk5Po2TrE7Gji4VReRESkzgrLq5mxLIOvsk8DcEfvVsy5txdBfhoTieupvIiISJ1sO1zI1MXp5JZU4utt5cU7ezDu2hiNiaTBqLyIiMhlcToN3v/7Qd5evw+H0yA2PJCFyUl0axVsdjRpYlReRETkJxWUVfHE0gy+3l8AwD2JrXl1RE8CbTqNSMPTb52IiFzSloNnmLYknfzSKvx8rLx8d09G9WmjMZGYRuVFREQuyOE0mL9xP/M27MdpQOeIZiwcn0SXyCCzo0kTp/IiIiL/Ib+0kseXZPDtwTMAjO7bhpfu6om/r5fJyURUXkRE5Ee+2V/A40vTKSirJsDXi1dH9OTepDZmxxKppfIiIiIA1DicvPvlfhZuOoBhQFxUEAuSk+gU0czsaCLnUXkRERFyiyuZuiSdbYcLAUju15YX7uiOn4/GRNL4qLyIiDRxX2XnM2PZDgrLq2lm8+a1e3txV3y02bFELkrlRUSkibI7nMz9Wza/+/shAHpEB7MgOYkOLQNNTiZyaSovIiJN0Imic0xdnM72o2cBeKB/O54Z3k1jInELKi8iIk3Ml3vzmLF8B8Xn7AT5efPGyN7c1quV2bFELpvKi4hIE1Fd4+SNdVn88ZvDAMS3CWH+uCTatggwOZlI3ai8iIg0ATmFFaQsTmdHThEA/3V9B2beFoevt9XcYCJXQOVFRMTDrdt9iqdW7KS0soYQfx/mjornlu6RZscSuWIqLyIiHqqqxsFrn2fy0ZajACS2DWX+uETaNNeYSNybyouIiAc6UlBOyuI0dp8oAeDRQbE8eWtXfLw0JhL3p/IiIuJhPtt5kpkf76KsqobmAT68PTqBm+IizI4lUm9UXkREPESl3cHLn+0ldesxAK5p35x54xJpFeJvcjKR+qXyIiLiAQ6eLmPyojSyckuxWOCxGzvyxJAueGtMJB5I5UVExM2tTj/BM6t2UVHtoEWgL++MSeBnXcLNjiXiMiovIiJu6ly1gxfX7GHpDzkAXBcbxryxiUQE+5mcTMS1VF5ERNzQ/rxSJqemsS+vDIsFpt7cmamDO+NltZgdTcTlVF5ERNzM8h9yeOGTPZyzOwgPsvHemAQGdGppdiyRBqPyIiLiJsqranj+k92sTDsBwMDOLXl7dALhQTaTk4k0LJUXERE3kJVbwuRFaRw8XY7VAtNv6cJjN3bCqjGRNEEqLyIijZhhGCz5PocX1+yhqsZJZLCNeWMT6RfbwuxoIqZReRERaaTKqmp4ZuUu1uw4CcCNXcN5a1Q8LZppTCRNm8qLiEgjtPtEMSmpaRw5U4GX1cJTQ7syaWCsxkQiqLyIiDQqhmHwv98d5ZXPMql2OIkO8WN+ciJ92oWZHU2k0VB5ERFpJEoq7cz8eCdrd+UCMKRbBHNHxRMa4GtyMpHGReVFRKQR2Hm8iMmpaeQUnsPHy8KvhsUx8YYOWCwaE4n8mMqLiIiJDMPgvzcfYc5fM7E7DNo092dBchIJMaFmRxNptFReRERMUlxh56kVO/jb3jwAhvWI4jf39SbE38fkZCKNm8qLiIgJ0o6dZUpqOieKzuHrZeXZ27vxQP92GhOJXAaVFxGRBuR0Gvzxm0O8sS6bGqdBuxYBLExOomfrELOjibgNqyvvvLCwkPHjxxMcHExoaCgTJ06krKzssrY1DIPbbrsNi8XC6tWrXRlTRKRBnC2v5uG//MBra7OocRrc0bsVn025QcVFpI5ceuVl/PjxnDp1ivXr12O323nooYeYNGkSqampP7ntu+++q8unIuIxvj9SyNTF6ZwqrsTX28rsO7uTfG1bPc6JXAGXlZfMzEzWrVvH999/T9++fQGYP38+w4cPZ+7cuURHR19024yMDN566y1++OEHWrVq5aqIIiIu53QavP/3g7y9fh8Op0Fsy0AWJCfRPTrY7Ggibstl5WXLli2EhobWFheAIUOGYLVa2bp1K/fcc88Ft6uoqCA5OZmFCxcSFRX1k/upqqqiqqqq9uuSkpKrDy8iUg8Kyqp4YmkGX+8vAOCexNa8OqIngTY93VDkarjsLyg3N5eIiIjzd+btTVhYGLm5uRfd7oknnmDAgAHcfffdl7WfOXPm8NJLL11VVhGR+rbl4BmmLUknv7QKPx8rL9/Vk1F922hMJFIP6vyE3ZkzZ2KxWC55y8rKuqIwa9asYePGjbz77ruXvc2sWbMoLi6uveXk5FzRvkVE6oPDafDel/sZ/8fvyC+tolNEM9ak3MDoa2JUXETqSZ2vvMyYMYMJEyZccp3Y2FiioqLIz88/b3lNTQ2FhYUXHQdt3LiRgwcPEhoaet7ykSNHMnDgQDZt2vQf29hsNmw2fTy8iJgvv7SSx5dk8O3BMwCM6tOGl+7uQYCvxkQi9anOf1Hh4eGEh4f/5Hr9+/enqKiI7du306dPH+Cf5cTpdNKvX78LbjNz5kwefvjh85b16tWLd955hzvvvLOuUUVEGsw3+wt4fGkGBWVVBPh68eqIntyb1MbsWCIeyWX/HejWrRvDhg3jkUce4YMPPsBut5OSksLYsWNrX2l04sQJBg8ezF/+8heuvfZaoqKiLnhVpm3btnTo0MFVUUVErliNw8l7G/az4KsDGAbERQWxIDmJThHNzI4m4rFcei1z0aJFpKSkMHjwYKxWKyNHjmTevHm137fb7WRnZ1NRUeHKGCIiLpFbXMnUJelsO1wIwLhr2zL7zu74+XiZnEzEs1kMwzDMDlGfSkpKCAkJobi4mOBgvY+CiLjGpux8pi/bQWF5NYG+XswZ2Zu74i/+/lUicml1OX/rWWQiInVgdzh5e/0+3t90EIDurYJZOD6JDi0DTU4m0nSovIiIXKaTReeYsjid7UfPAvBA/3Y8M7ybxkQiDUzlRUTkMny5N48nV+ygqMJOkM2b39zXm+G99PElImZQeRERuYTqGidvrMvij98cBqB3mxAWjEuibYsAk5OJNF0qLyIiF5FTWEHK4nR25BQB8F/Xd+BXt3XF5q0xkYiZVF5ERC5g3e5cnl6xg5LKGoL9vJk7Kp5be/z0h8WKiOupvIiI/B9VNQ7mrM3iw2+PAJDYNpT54xJp01xjIpHGQuVFRORfjp4pJyU1nV0nigF49GexPDm0Kz5edf4MWxFxIZUXERHg852nmPnxTkqramge4MNbo+O5OS7S7FgicgEqLyLSpFXaHbz6+V7+97tjAFzTvjnzxiXSKsTf5GQicjEqLyLSZB06Xcbk1HQyT5UA8NiNHZl+Sxe8NSYSadRUXkSkSVqdfoJnVu2iotpBi0Bf3h6TwKAu4WbHEpHLoPIiIk3KuWoHL67Zw9IfcgC4LjaM98YmEhnsZ3IyEblcKi8i0mQcyC9l8qJ0svNKsVhgys2dmTa4M15Wi9nRRKQOVF5EpElYsf04z6/ezTm7g/AgG++NSWBAp5ZmxxKRK6DyIiIeraK6hudW72Zl2gkAbujUknfGJBAeZDM5mYhcKZUXEfFYWbklTF6UxsHT5VgtMP2WLvzyxk4aE4m4OZUXEfE4hmGw9PscZq/ZQ1WNk8hgG/PGJtIvtoXZ0USkHqi8iIhHKauq4dlVu/gk4yQAg7qE8/boeFo005hIxFOovIiIx9hzspiU1HQOF5TjZbXw5K1defRnsVg1JhLxKCovIuL2DMPgf7ce45XP9lJd4yQ6xI/5yYn0aRdmdjQRcQGVFxFxayWVdmZ9vIvPd50CYEi3CN68L57mgb4mJxMRV1F5ERG3tfN4ESmp6RwrrMDbamHmbXFMvKEDFovGRCKeTOVFRNyOYRh8+O0RXlubid1h0DrUnwXJiSS2bW52NBFpACovIuJWiivsPLViB3/bmwfA0B6RvDEynpAAH5OTiUhDUXkREbeRfuwsKanpnCg6h6+XlWdv78YD/dtpTCTSxKi8iEijZxgGf/z6ML9Zl0WN06BdiwAWjEuiV5sQs6OJiAlUXkSkUTtbXs2Ty3ewISsfgNt7t2LOvb0I9tOYSKSpUnkRkUbrhyOFTFmczqniSny9rbxwR3fG92urMZFIE6fyIiKNjtNp8ME/DvLW3/bhcBrEtgxkQXIS3aODzY4mIo2AyouINCoFZVVMX7aDf+w7DcCIhGhevacXzWx6uBKRf9KjgYg0Gt8dOsPUxenkl1bh52Plpbt6MLpvjMZEInIelRcRMZ3DabDwqwO8++U+nAZ0imjGwuQkukYFmR1NRBohlRcRMVV+aSVPLM1g84EzANzXpw0v392DAF89PInIhenRQURMs/lAAdOWZFBQVoW/jxevjujJyD5tzI4lIo2cyouINDiH0+C9L/cx/6sDGAZ0jQxi4fgkOkU0MzuaiLgBlRcRaVB5JZVMXZzO1sOFAIy7NobZd/bAz8fL5GQi4i5UXkSkwfx932meWJpBYXk1gb5evHZvL+5OaG12LBFxMyovIuJyNQ4nb63fx/ubDgLQvVUwC5ITiQ3XmEhE6k7lRURc6mTROaYuTueHo2cBuP+6djx7ezeNiUTkiqm8iIjLbMzKY/qyHRRV2AmyefP6yN7c3ruV2bFExM2pvIhIvauucfLmF1n84evDAPRqHcKC5ETatQg0OZmIeAKVFxGpVzmFFUxZnE5GThEAD13fnpm3xWHz1phIROqHyouI1Jsv9uTy1PIdlFTWEOznzZuj4hnaI8rsWCLiYVReROSqVdU4mLM2iw+/PQJAQkwoC5ITadM8wNxgIuKRVF5E5KocPVNOSmo6u04UAzDpZ7E8NbQrPl5Wk5OJiKdSeRGRK/b5zlPM/HgnpVU1hAb48PboeG6OizQ7loh4OJUXEamzSruDVz/fy/9+dwyAvu2aM29cItGh/iYnE5GmQOVFROrkcEE5kxelsfdUCQCP3diR6bd0wVtjIhFpICovInLZPsk4wTMrd1Fe7aBFoC9vj0lgUJdws2OJSBOj8iIiP+lctYOXPt3Dku9zAOjXIYx54xKJDPYzOZmINEUqLyJySQfyS5m8KJ3svFIsFphyc2em3txJYyIRMY3Ki4hc1Irtx3l+9W7O2R20bGbjvbEJXN+ppdmxRKSJU3kRkf9QUV3D86v38HHacQCu79SCd8YkEBGkMZGImE/lRUTOk51byuTUNA7kl2G1wBNDuvDYTZ3wslrMjiYiAqi8iMi/GIbBsh9yeOGTPVTVOIkMtvHe2ESui21hdjQRkfO47Bl3hYWFjB8/nuDgYEJDQ5k4cSJlZWU/ud2WLVu4+eabCQwMJDg4mJ/97GecO3fOVTFFBCirquGJpRn86uNdVNU4GdQlnLVTB6q4iEij5LIrL+PHj+fUqVOsX78eu93OQw89xKRJk0hNTb3oNlu2bGHYsGHMmjWL+fPn4+3tzY4dO7Ba9aoGEVfZe7KElNQ0DhWU42W18OStXXn0Z7FYNSYSkUbKYhiGUd93mpmZSffu3fn+++/p27cvAOvWrWP48OEcP36c6OjoC2533XXXccstt/DKK69c8b5LSkoICQmhuLiY4ODgK74fEU9nGAaLth7j5c/2Ul3jpFWIH/PHJdK3fZjZ0USkCarL+dsllzS2bNlCaGhobXEBGDJkCFarla1bt15wm/z8fLZu3UpERAQDBgwgMjKSQYMG8c0331xyX1VVVZSUlJx3E5FLK6m0k7I4nedW76a6xsnguAjWTh2o4iIibsEl5SU3N5eIiIjzlnl7exMWFkZubu4Ftzl06BAAL774Io888gjr1q0jKSmJwYMHs3///ovua86cOYSEhNTeYmJi6u8HEfFAu44Xc8e8b/h85ym8rRaeu70bf3ywL80Dfc2OJiJyWepUXmbOnInFYrnkLSsr64qCOJ1OAB599FEeeughEhMTeeedd+jatSt//vOfL7rdrFmzKC4urr3l5ORc0f5FPJ1hGHy4+TAj3/+WY4UVtA71Z/kv+vPwwFgsFj2/RUTcR52esDtjxgwmTJhwyXViY2OJiooiPz//vOU1NTUUFhYSFRV1we1atWoFQPfu3c9b3q1bN44dO3bR/dlsNmw222WkF2m6iivsPP3xDr7YkwfArd0jefO+eEICfExOJiJSd3UqL+Hh4YSH//QnyPbv35+ioiK2b99Onz59ANi4cSNOp5N+/fpdcJv27dsTHR1Ndnb2ecv37dvHbbfdVpeYIvJ/pB87y5TF6Rw/ew5fLyvPDI/jwQHtdbVFRNyWS57z0q1bN4YNG8YjjzzCtm3b2Lx5MykpKYwdO7b2lUYnTpwgLi6Obdu2AWCxWHjqqaeYN28eK1as4MCBAzz//PNkZWUxceJEV8QU8WiGYfDHrw8x6oMtHD97jrZhAXz8ywFMuL6DiouIuDWXvc/LokWLSElJYfDgwVitVkaOHMm8efNqv2+328nOzqaioqJ22eOPP05lZSVPPPEEhYWFxMfHs379ejp27OiqmCIe6Wx5NU8u38GGrH+Ob2/v1Yo5I3sR7KcxkYi4P5e8z4uZ9D4v0tRtP1rIlNR0ThZX4utt5YU7ujO+X1tdbRGRRq0u5299tpGIh3A6DX73j0PM/Vs2DqdBh5aBLEhOpEd0iNnRRETqlcqLiAc4U1bF9GU7+Pu+0wDcnRDNr+/pRTOb/sRFxPPokU3EzW09dIapS9LJK6nC5m3l5bt7MLpvjMZEIuKxVF5E3JTDafDbrw7wzpf7cBrQMTyQ347vQ9eoILOjiYi4lMqLiBs6XVrF40vT2XzgDAAjk9rwyogeBPjqT1pEPJ8e6UTczOYDBUxbkkFBWRX+Pl68MqIn9/VpY3YsEZEGo/Ii4iYcToP3Nuxn/sb9GAZ0jQxi4fhEOkVoTCQiTYvKi4gbyCupZNqSdL47VAjA2GtimH1nD/x9vUxOJiLS8FReRBq5v+87zfSlGZwprybQ14vX7u3F3QmtzY4lImIalReRRqrG4eTt9fv47aaDAHRrFczC5ERiw5uZnExExFwqLyKN0Knic0xdnM73R84C8PPr2vLc7d3x89GYSERE5UWkkdmYlceMZTs4W2EnyObNnJG9uKN3tNmxREQaDZUXkUbC7nDy5hfZ/P4fhwDo1TqEBcmJtGsRaHIyEZHGReVFpBE4fraClNR0MnKKAJgwoD2zhsdh89aYSETkx1ReREz2xZ5cnlq+g5LKGoL9vHnjvniG9YwyO5aISKOl8iJikuoaJ3P+msl/bz4CQHxMKAvGJRITFmBuMBGRRk7lRcQEx85UkLI4jZ3HiwF4ZGAHnhoah6+31eRkIiKNn8qLSANbu+sUv1qxk9KqGkIDfHhrVDyDu0WaHUtExG2ovIg0kEq7g19/nsn/fHcUgL7tmjNvXCLRof4mJxMRcS8qLyIN4HBBOZMXpbH3VAkAv7yxI9Nv6YKPl8ZEIiJ1pfIi4mKfZJzgmZW7KK92EBboy9uj47mxa4TZsURE3JbKi4iLVNodvPTpHhZvywHg2g5hzBubSFSIn8nJRETcm8qLiAscyC8jJTWNrNxSLBaYclMnpg7ujLfGRCIiV03lRaSefbz9OM+t3s05u4OWzWy8OyaBGzq3NDuWiIjHUHkRqScV1TW88MkeVmw/DsCAji14d2wCEUEaE4mI1CeVF5F6sC+vlMmL0tifX4bVAo8P6cLkmzrhZbWYHU1ExOOovIhcBcMwWPZDDrPX7KHS7iQiyMa8cYlcF9vC7GgiIh5L5UXkCpVV1fDcql2szjgJwM+6hPP26HhaNrOZnExExLOpvIhcgb0nS0hJTeNQQTleVgszbu3CL37WEavGRCIiLqfyIlIHhmGQuu0YL326l+oaJ61C/Jg3LpFr2oeZHU1EpMlQeRG5TKWVdmau3MXnO08BcHNcBG+Niqd5oK/JyUREmhaVF5HLsPtEMZNT0zh6pgJvq4VfDYtj4g0dNCYSETGByovIJRiGwUffHuG1tVlUO5y0DvVnfnIiSW2bmx1NRKTJUnkRuYjic3Z+tWIn6/bkAnBr90jevC+ekAAfk5OJiDRtKi8iF5CRU0RKahrHz57Dx8vCM8O7MWFAeywWjYlERMym8iLyfxiGwZ++Oczrf82ixmnQNiyABcmJ9G4TanY0ERH5F5UXkX8pqqjmyeU7+DIzH4DhvaJ4fWRvgv00JhIRaUxUXkSA7UcLmZKazsniSny9rTx/R3d+3q+txkQiIo2Qyos0aU6nwe+/PsSbX2TjcBp0aBnIguREekSHmB1NREQuQuVFmqwzZVXMWL6DTdmnAbgrPprX7u1FM5v+LEREGjM9SkuTtPXQGaYuSSevpAqbt5WX7urBmGtiNCYSEXEDKi/SpDidBr/ddIC31+/DaUDH8EAWjk8iLirY7GgiInKZVF6kyThdWsX0ZRl8vb8AgHuTWvPK3T0J1JhIRMSt6FFbmoRvDxQwbWkGp0ur8Pfx4uW7ezCqb4zZsURE5AqovIhHczgN3tuwn/kb92MY0CWyGQuTk+gcGWR2NBERuUIqL+Kx8koqmbYkne8OFQIw9poYZt/ZA39fL5OTiYjI1VB5EY/0j32neWJpBmfKqwn09eK1e3txd0Jrs2OJiEg9UHkRj1LjcPLOl/v47aaDGAZ0axXMwuREYsObmR1NRETqicqLeIxTxeeYujid74+cBWB8v7Y8f0d3/Hw0JhIR8SQqL+IRvsrKZ/qyDM5W2Glm8+b1kb24o3e02bFERMQFVF7ErdkdTuZ+kc3v/nEIgJ6tg1mYnES7FoEmJxMREVdReRG3dfxsBVMWp5N+rAiACQPaM2t4HDZvjYlERDyZyou4pb/tyeWpFTspPmcnyM+bN+/rzbCercyOJSIiDUDlRdxKdY2TOX/N5L83HwEgPiaUBeMSiQkLMDeYiIg0GJUXcRvHzlSQsjiNnceLAXhkYAeeGhqHr7fV5GQiItKQVF7ELfx11ymeXrGT0qoaQgN8mHtfPEO6R5odS0RETOCy/7IWFhYyfvx4goODCQ0NZeLEiZSVlV1ym9zcXO6//36ioqIIDAwkKSmJjz/+2FURxQ1U2h288MlufrkojdKqGvq0a87nUwequIiINGEuu/Iyfvx4Tp06xfr167Hb7Tz00ENMmjSJ1NTUi27zwAMPUFRUxJo1a2jZsiWpqamMHj2aH374gcTERFdFlUbqcEE5Kalp7DlZAsAvBnVkxq1d8PHSmEhEpCmzGIZh1PedZmZm0r17d77//nv69u0LwLp16xg+fDjHjx8nOvrCbx7WrFkz3n//fe6///7aZS1atOA3v/kNDz/88GXtu6SkhJCQEIqLiwkODr76H0ZMsWbHSZ5ZuYuyqhrCAn15e3Q8N3aNMDuWiIi4SF3O3y75L+yWLVsIDQ2tLS4AQ4YMwWq1snXr1otuN2DAAJYuXUphYSFOp5MlS5ZQWVnJjTfeeNFtqqqqKCkpOe8m7qvS7mDWyl1MXZxOWVUN13YIY+3UgSouIiJSyyVjo9zcXCIizj/ZeHt7ExYWRm5u7kW3W7ZsGWPGjKFFixZ4e3sTEBDAqlWr6NSp00W3mTNnDi+99FK9ZRfzHMgvIyU1jazcUiwWSLmpE9MGd8ZbYyIREfk/6nRWmDlzJhaL5ZK3rKysKw7z/PPPU1RUxJdffskPP/zA9OnTGT16NLt27broNrNmzaK4uLj2lpOTc8X7F/OsTDvOXQu+ISu3lJbNfPmf/+rHjFu7qriIiMh/qNOVlxkzZjBhwoRLrhMbG0tUVBT5+fnnLa+pqaGwsJCoqKgLbnfw4EEWLFjA7t276dGjBwDx8fF8/fXXLFy4kA8++OCC29lsNmw2W11+DGlEKqprmP3JHpZvPw7AgI4teHdMAhHBfiYnExGRxqpO5SU8PJzw8PCfXK9///4UFRWxfft2+vTpA8DGjRtxOp3069fvgttUVFQAYLWe/z9tLy8vnE5nXWKKm9iXV8rkRWnszy/DaoFpg7uQcnMnvKwWs6OJiEgj5pJr8t26dWPYsGE88sgjbNu2jc2bN5OSksLYsWNrX2l04sQJ4uLi2LZtGwBxcXF06tSJRx99lG3btnHw4EHeeust1q9fz4gRI1wRU0xiGAbLvs/hrgXfsD+/jIggG4sevo5pQzqruIiIyE9y2fu8LFq0iJSUFAYPHozVamXkyJHMmzev9vt2u53s7OzaKy4+Pj6sXbuWmTNncuedd1JWVkanTp346KOPGD58uKtiSgMrr6rhudW7WZV+AoCBnVvyzpgEWjbT6E9ERC6PS97nxUx6n5fGK/NUCZMXpXGooBwvq4Xpt3Thl4M6YtXVFhGRJq8u5299tpG4nGEYpG47xkuf7qW6xklUsB/zkxO5pn2Y2dFERMQNqbyIS5VW2pm1chef7TwFwM1xEcwdFU9YoK/JyURExF2pvIjL7D5RTEpqGkfOVOBttfD0sK48fEOsxkQiInJVVF6k3hmGwV+2HOXXn2dS7XDSOtSf+cmJJLVtbnY0ERHxACovUq+Kz9mZ+fFO/rr7nx8DcUv3SN68rzehARoTiYhI/VB5kXqTkVNESmoax8+ew8fLwqzbuvHQ9e2xWDQmEhGR+qPyIlfNMAz+9M1hfrMuC7vDICbMnwXjkoiPCTU7moiIeCCVF7kqRRXVPLl8J19m5gEwvFcUr4/sTbCfj8nJRETEU6m8yBXbfvQsU1LTOFlcia+Xlefv6MbPr2unMZGIiLiUyovUmdNp8PuvD/HmF9k4nAbtWwSwIDmJnq1DzI4mIiJNgMqL1ElheTXTl2WwKfs0AHfFR/Pavb1oZtOvkoiINAydceSybTtcyNTF6eSWVGLztvLiXT0Ye02MxkQiItKgVF7kJzmdBr/ddIC31+/DaUBseCALk5Po1koffCkiIg1P5UUu6XRpFdOXZfD1/gIA7k1szSsjehKoMZGIiJhEZyC5qG8PFjBtSQanS6vw87Hyyt09GdU3xuxYIiLSxKm8yH9wOA3mb9zPvA37cRrQJbIZC5OT6BwZZHY0ERERlRc5X35JJdOWZLDl0BkAxvSN4cW7euDv62VyMhERkX9SeZFaX+8/zRNLMygoqybA14vX7unFiMTWZscSERE5j8qLUONw8u6X+1m46QCGAXFRQSwcn0TH8GZmRxMREfkPKi9N3Knic0xbnMG2I4UAjO/Xlufv6I6fj8ZEIiLSOKm8NGFfZeUzfVkGZyvsNLN5M+feXtwZH212LBERkUtSeWmC7A4nc7/I5nf/OARAz9bBLBiXRPuWgSYnExER+WkqL03MiaJzTElNI+1YEQATBrRn1vA4bN4aE4mIiHtQeWlC1u/N48nlOyg+ZyfIz5s37+vNsJ6tzI4lIiJSJyovTUB1jZPfrMviT98cBiC+TQgLkpOICQswOZmIiEjdqbx4uJzCClJS09hxvBiAiTd04FfD4vD1tpqcTERE5MqovHiwdbtP8dSKnZRW1hDi78PcUfHc0j3S7FgiIiJXReXFA1XaHcxZm8lHW44CkNQ2lPnJSbQO9Tc5mYiIyNVTefEwRwrKmZyaxp6TJQA8OiiWJ2/tio+XxkQiIuIZVF48yKc7TjJr5S7KqmoIC/TlrdHx3NQ1wuxYIiIi9UrlxQNU2h28/NleUrceA+Da9mHMG5dIVIifyclERETqn8qLmzt4uozJi9LIyi3FYoGUmzoxbXBnvDUmEhERD6Xy4sZWpR/n2VW7qah20LKZL++MSWBg53CzY4mIiLiUyosbOlftYPaa3Sz74TgA/WNb8N7YBCKCNSYSERHPp/LiZvbnlfLYojT255dhscC0wZ2ZcnNnvKwWs6OJiIg0CJUXN2EYBsu3H+eFT3ZTaXcSHmTjvbEJDOjY0uxoIiIiDUrlxQ2UV9Xw/OrdrEw/AcDAzi15Z0wCLZvZTE4mIiLS8FReGrnMUyVMTk3j0OlyrBaYcWtXfjmoI1aNiUREpIlSeWmkDMNg8bYcXvp0D1U1TqKC/Zg3LpFrO4SZHU1ERMRUKi+NUGmlnWdW7ebTHScBuKlrOG+NTiAs0NfkZCIiIuZTeWlkdp8oJiU1jSNnKvC2WnhqaFceGRirMZGIiMi/qLw0EoZh8D/fHeXVzzKpdjhpHerPvHGJ9GnX3OxoIiIijYrKSyNQfM7OrJU7WbsrF4Ah3SKZO6o3oQEaE4mIiPyYyovJduQUkbI4jZzCc/h4WZh5Wzf+6/r2WCwaE4mIiFyIyotJDMPgz5uP8PpfM7E7DGLC/FkwLon4mFCzo4mIiDRqKi8mKKqo5snlO/kyMw+A23pG8frI3oT4+5icTEREpPFTeWlg24+eZeridE4UncPXy8pzd3Tj/uvaaUwkIiJymVReGojTafCHrw/x5hfZ1DgN2rcIYEFyEj1bh5gdTURExK2ovDSAwvJqZizL4Kvs0wDcGR/Na/f0JMhPYyIREZG6UnlxsW2HC5m6OJ3ckkps3lZm39mDcdfGaEwkIiJyhVReXMTpNHj/7wd5e/0+HE6D2PBAFiYn0a1VsNnRRERE3JrKiwsUlFXxxNIMvt5fAMC9ia15ZURPAm063CIiIldLZ9N6tuXgGaYtSSe/tAo/Hysv392TUX3aaEwkIiJST1Re6onDaTB/437mbdiP04DOEc1YOD6JLpFBZkcTERHxKCov9SC/tJLHl2Tw7cEzAIzu24aX7uqJv6+XyclEREQ8j8rLVfpmfwGPL02noKyaAF8vXh3Rk3uT2pgdS0RExGNZXXXHv/71rxkwYAABAQGEhoZe1jaGYfDCCy/QqlUr/P39GTJkCPv373dVxKtS43Ay94ts7v/zVgrKqomLCmJNyg0qLiIiIi7msvJSXV3NqFGj+OUvf3nZ27zxxhvMmzePDz74gK1btxIYGMjQoUOprKx0VcwrkltcSfIftrLgqwMYBiT3a8vqydfTKaKZ2dFEREQ8nsUwDMOVO/jwww95/PHHKSoquuR6hmEQHR3NjBkzePLJJwEoLi4mMjKSDz/8kLFjx17W/kpKSggJCaG4uJjg4Pp/T5WvsvOZsWwHheXVNLN589q9vbgrPrre9yMiItKU1OX83Wie83L48GFyc3MZMmRI7bKQkBD69evHli1bLlpeqqqqqKqqqv26pKTEJfnsDidz/5bN7/5+CIAe0cEsTE6ifctAl+xPRERELsxlY6O6ys3NBSAyMvK85ZGRkbXfu5A5c+YQEhJSe4uJiXFJvg2ZebXF5cH+7fj4lwNUXERERExQp/Iyc+ZMLBbLJW9ZWVmuynpBs2bNori4uPaWk5Pjkv0M7RHFz69ry/vjk3jp7p74+ehl0CIiImao09hoxowZTJgw4ZLrxMbGXlGQqKgoAPLy8mjVqlXt8ry8PBISEi66nc1mw2azXdE+68JisfDqiF4u34+IiIhcWp3KS3h4OOHh4S4J0qFDB6KiotiwYUNtWSkpKWHr1q11esWSiIiIeDaXPefl2LFjZGRkcOzYMRwOBxkZGWRkZFBWVla7TlxcHKtWrQL+eWXj8ccf59VXX2XNmjXs2rWLBx54gOjoaEaMGOGqmCIiIuJmXPZqoxdeeIGPPvqo9uvExEQAvvrqK2688UYAsrOzKS4url3n6aefpry8nEmTJlFUVMQNN9zAunXr8PPzc1VMERERcTMuf5+Xhubq93kRERGR+leX83ejeam0iIiIyOVQeRERERG3ovIiIiIibkXlRURERNyKyouIiIi4FZUXERERcSsqLyIiIuJWVF5ERETErai8iIiIiFtx2ccDmOXfbxhcUlJichIRERG5XP8+b1/OG/97XHkpLS0FICYmxuQkIiIiUlelpaWEhIRcch2P+2wjp9PJyZMnCQoKwmKx1Ot9l5SUEBMTQ05Ojj43yYV0nBuGjnPD0HFuODrWDcNVx9kwDEpLS4mOjsZqvfSzWjzuyovVaqVNmzYu3UdwcLD+MBqAjnPD0HFuGDrODUfHumG44jj/1BWXf9MTdkVERMStqLyIiIiIW1F5qQObzcbs2bOx2WxmR/FoOs4NQ8e5Yeg4Nxwd64bRGI6zxz1hV0RERDybrryIiIiIW1F5EREREbei8iIiIiJuReVFRERE3IrKy48sXLiQ9u3b4+fnR79+/di2bdsl11++fDlxcXH4+fnRq1cv1q5d20BJ3VtdjvMf/vAHBg4cSPPmzWnevDlDhgz5yX8X+ae6/j7/25IlS7BYLIwYMcK1AT1EXY9zUVERkydPplWrVthsNrp06aLHjstU12P97rvv0rVrV/z9/YmJieGJJ56gsrKygdK6n3/84x/ceeedREdHY7FYWL169U9us2nTJpKSkrDZbHTq1IkPP/zQ5TkxpNaSJUsMX19f489//rOxZ88e45FHHjFCQ0ONvLy8C66/efNmw8vLy3jjjTeMvXv3Gs8995zh4+Nj7Nq1q4GTu5e6Hufk5GRj4cKFRnp6upGZmWlMmDDBCAkJMY4fP97Ayd1LXY/zvx0+fNho3bq1MXDgQOPuu+9umLBurK7Huaqqyujbt68xfPhw45tvvjEOHz5sbNq0ycjIyGjg5O6nrsd60aJFhs1mMxYtWmQcPnzY+OKLL4xWrVoZTzzxRAMndx9r1641nn32WWPlypUGYKxateqS6x86dMgICAgwpk+fbuzdu9eYP3++4eXlZaxbt86lOVVe/o9rr73WmDx5cu3XDofDiI6ONubMmXPB9UePHm3cfvvt5y3r16+f8eijj7o0p7ur63H+sZqaGiMoKMj46KOPXBXRI1zJca6pqTEGDBhg/PGPfzQefPBBlZfLUNfj/P777xuxsbFGdXV1Q0X0GHU91pMnTzZuvvnm85ZNnz7duP76612a01NcTnl5+umnjR49epy3bMyYMcbQoUNdmMwwNDb6l+rqarZv386QIUNql1mtVoYMGcKWLVsuuM2WLVvOWx9g6NChF11fruw4/1hFRQV2u52wsDBXxXR7V3qcX375ZSIiIpg4cWJDxHR7V3Kc16xZQ//+/Zk8eTKRkZH07NmT1157DYfD0VCx3dKVHOsBAwawffv22tHSoUOHWLt2LcOHD2+QzE2BWedBj/tgxitVUFCAw+EgMjLyvOWRkZFkZWVdcJvc3NwLrp+bm+uynO7uSo7zj/3qV78iOjr6P/5g5P+7kuP8zTff8Kc//YmMjIwGSOgZruQ4Hzp0iI0bNzJ+/HjWrl3LgQMHeOyxx7Db7cyePbshYrulKznWycnJFBQUcMMNN2AYBjU1NfziF7/gmWeeaYjITcLFzoMlJSWcO3cOf39/l+xXV17Erbz++ussWbKEVatW4efnZ3Ycj1FaWsr999/PH/7wB1q2bGl2HI/mdDqJiIjg97//PX369GHMmDE8++yzfPDBB2ZH8zibNm3itdde47e//S1paWmsXLmSzz//nFdeecXsaHKVdOXlX1q2bImXlxd5eXnnLc/LyyMqKuqC20RFRdVpfbmy4/xvc+fO5fXXX+fLL7+kd+/erozp9up6nA8ePMiRI0e48847a5c5nU4AvL29yc7OpmPHjq4N7Yau5Pe5VatW+Pj44OXlVbusW7du5ObmUl1dja+vr0szu6srOdbPP/88999/Pw8//DAAvXr1ory8nEmTJvHss89iter/71frYufB4OBgl111AV15qeXr60ufPn3YsGFD7TKn08mGDRvo37//Bbfp37//eesDrF+//qLry5UdZ4A33niDV155hXXr1tG3b9+GiOrW6nqc4+Li2LVrFxkZGbW3u+66i5tuuomMjAxiYmIaMr7buJLf5+uvv54DBw7UlkOAffv20apVKxWXS7iSY11RUfEfBeXfpdHQx/rVC9POgy59OrCbWbJkiWGz2YwPP/zQ2Lt3rzFp0iQjNDTUyM3NNQzDMO6//35j5syZtetv3rzZ8Pb2NubOnWtkZmYas2fP1kulL0Ndj/Prr79u+Pr6GitWrDBOnTpVeystLTXrR3ALdT3OP6ZXG12euh7nY8eOGUFBQUZKSoqRnZ1tfPbZZ0ZERITx6quvmvUjuI26HuvZs2cbQUFBxuLFi41Dhw4Zf/vb34yOHTsao0ePNutHaPRKS0uN9PR0Iz093QCMt99+20hPTzeOHj1qGIZhzJw507j//vtr1//3S6WfeuopIzMz01i4cKFeKm2G+fPnG23btjV8fX2Na6+91vjuu+9qvzdo0CDjwQcfPG/9ZcuWGV26dDF8fX2NHj16GJ9//nkDJ3ZPdTnO7dq1M4D/uM2ePbvhg7uZuv4+/18qL5evrsf522+/Nfr162fYbDYjNjbW+PWvf23U1NQ0cGr3VJdjbbfbjRdffNHo2LGj4efnZ8TExBiPPfaYcfbs2YYP7ia++uqrCz7e/vu4Pvjgg8agQYP+Y5uEhATD19fXiI2NNf77v//b5TkthqFrZyIiIuI+9JwXERERcSsqLyIiIuJWVF5ERETErai8iIiIiFtReRERERG3ovIiIiIibkXlRURERNyKyouIiIi4FZUXERERcSsqLyIiIuJWVF5ERETErai8iIiIiFv5f0lAL9XHsFbXAAAAAElFTkSuQmCC\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(output_probs, nll_grad(output_probs, 1))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can now write the training loop using sigmoid activation and negative log likelihood:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 train loss: 0.2682927454493638\n",
      "Valid loss: 0.26448187252356126\n",
      "Epoch 10 train loss: 0.10106777344637874\n",
      "Valid loss: 0.12040902930724136\n",
      "Epoch 20 train loss: 0.076515962768458\n",
      "Valid loss: 0.09385188397071516\n",
      "Epoch 30 train loss: 0.06805932971500155\n",
      "Valid loss: 0.09548004093000301\n",
      "Epoch 40 train loss: 0.0624920649725441\n",
      "Valid loss: 0.10128727568213479\n"
     ]
    }
   ],
   "source": [
    "net = ClassificationNet()\n",
    "lr = 1e-2\n",
    "epochs = 50\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    epoch_loss = 0\n",
    "    for x, target in zip(train_x, train_y):\n",
    "        x = x.reshape(1,-1)\n",
    "        # Run the sigmoud function over the output of the neural network\n",
    "        pred = sigmoid(net.forward(x))\n",
    "\n",
    "        # Compute the gradient using the nll_grad function\n",
    "        grad = nll_grad(pred, target)\n",
    "        epoch_loss += nll(pred, target)[0,0]\n",
    "\n",
    "        # Update the network parameters\n",
    "        net.backward(grad, lr)\n",
    "\n",
    "    if epoch % 10 == 0:\n",
    "        print(f\"Epoch {epoch} train loss: {epoch_loss / len(train_x)}\")\n",
    "        epoch_loss = 0\n",
    "        for x, target in zip(valid_x, valid_y):\n",
    "            x = x.reshape(1,-1)\n",
    "            pred = sigmoid(net.forward(x))\n",
    "            epoch_loss += nll(pred, target)[0,0]\n",
    "        print(f\"Valid loss: {epoch_loss / len(valid_x)}\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see above, we were able to reduce the loss from epoch to epoch and fit the data fairly well by the end of training."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Classification With Multiple Targets\n",
    "\n",
    "Now that we've covered binary classification, let's cover classification with multiple targets.  Let's say that our telescope can id `3` different kinds of objects, and we want to know the probability that it has detected each one.  These 3 objects are stars, galaxies, and quasars.\n",
    "\n",
    "We can't use binary classification, since that can only tell us if a single object is present or not.  We'll instead need to use multi-class classification.\n",
    "\n",
    "Our first step is to encode our targets into one-hot vectors.  Each position in the vector will correspond to a single class.  We'll say that a star corresponds to position `0`, a galaxy to position `1`, and a quasar to position `2`:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "outputs": [],
   "source": [
    "def encode(target, max_value=3):\n",
    "    encoded = np.zeros((1,max_value))\n",
    "    encoded[0,target] = 1\n",
    "    return encoded"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 0. 0.]]\n",
      "[[0. 1. 0.]]\n",
      "[[0. 0. 1.]]\n"
     ]
    }
   ],
   "source": [
    "# Encode a star\n",
    "print(encode(0))\n",
    "\n",
    "# Encode a galaxy\n",
    "print(encode(1))\n",
    "\n",
    "# Encode a quasar\n",
    "print(encode(2))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see above, we end up with a 3-element vector after encoding.  The position in the vector that is `1` corresponds to our class label (star, galaxy, or quasar).  This is why it is called one-hot encoding - only one position in the vector is non-zero.\n",
    "\n",
    "We'll use our network to predict this vector by outputting 3 number.  Similar to what we did with the sigmoid function, we need a way for the network to output values between `0` and `1` for each position in the vector.\n",
    "\n",
    "We also need all of the elements in the output vector to add to `1`.  Each vector element is the probability that the item belongs to that class.  Our total probability has to be `100%`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Softmax Function\n",
    "\n",
    "The softmax function will normalize the values in our output vector to meet the criteria above:\n",
    "\n",
    "$\\zeta=\\frac{e^{\\hat{y_{i}}}}{\\sum_{i=j}e^{\\hat{y_{j}}}}$\n",
    "\n",
    "Confusingly, the softmax and sigmoid functions usually the same symbol to define them.  To eliminate that confusion, we'll use $\\zeta$ for softmax.  Intuitively, the softmax will raise the current prediction to the power $e^{y_{i}}$, then divide by the sum of all the other predictions.  Larger predictions will get pushed closer to `1`, and smaller predictions will get pushed closer to `0`.  This is why it is called a softmax - it is a \"softer\" way of taking the maximum of the vector.\n",
    "\n",
    "Let's take a look at the function:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "outputs": [],
   "source": [
    "def softmax(preds):\n",
    "    # Raise e to the power of each of the predictions\n",
    "    preds = np.exp(preds)\n",
    "    # Divide the predictions by the sum\n",
    "    return preds / np.sum(preds)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "outputs": [
    {
     "data": {
      "text/plain": "array([2.06106005e-09, 4.53978686e-05, 9.99954600e-01])"
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "softmax(np.array([10,20,30]))"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see above, softmax pushed `30` very close to `1` in the output vector, and pushed the other 2 values close to `0`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Log Loss\n",
    "\n",
    "We use log loss as our loss metric."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "outputs": [],
   "source": [
    "def log_loss(predicted, actual):\n",
    "    tol = 1e-6\n",
    "    cross_entropy = actual * np.log(predicted + tol)\n",
    "    return -np.sum(cross_entropy)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Log loss gradient\n",
    "\n",
    "Luckily, the gradient of log loss and the softmax end up with the same gradient as sigmoid and negative log likelihood."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "outputs": [],
   "source": [
    "log_loss_grad = lambda pred, actual: pred - actual"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Training Loop - Multiple Classes"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "outputs": [],
   "source": [
    "from csv_data import SkyServerDatasetWrapper\n",
    "\n",
    "\n",
    "wrapper = SkyServerDatasetWrapper(\"cpu\")\n",
    "[train_x, train_y], [valid_x, valid_y], [test_x, test_y] = wrapper.get_flat_datasets()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 train loss: 0.5340438424762264\n",
      "Valid loss: 0.369248679718005\n",
      "Epoch 10 train loss: 0.17076418611771466\n",
      "Valid loss: 0.2060768028825078\n",
      "Epoch 20 train loss: 0.12944873470801835\n",
      "Valid loss: 0.16616614005982078\n",
      "Epoch 30 train loss: 0.10977823237985218\n",
      "Valid loss: 0.14806029236884094\n",
      "Epoch 40 train loss: 0.09653654780216854\n",
      "Valid loss: 0.13613702515812218\n"
     ]
    }
   ],
   "source": [
    "net = ClassificationNet(3)\n",
    "lr = 1e-3\n",
    "epochs = 50\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    epoch_loss = 0\n",
    "    for x, target in zip(train_x, train_y):\n",
    "        x = x.reshape(1,-1)\n",
    "        # Run the softmax function over the output of the neural network\n",
    "        pred = softmax(net.forward(x))\n",
    "\n",
    "        target = encode(target)\n",
    "        # Compute the gradient using the nll_grad function\n",
    "        grad = log_loss_grad(pred, target)\n",
    "        epoch_loss += log_loss(pred, target)\n",
    "\n",
    "        # Update the network parameters\n",
    "        net.backward(grad, lr)\n",
    "\n",
    "    if epoch % 10 == 0:\n",
    "        print(f\"Epoch {epoch} train loss: {epoch_loss / len(train_x)}\")\n",
    "        epoch_loss = 0\n",
    "        for x, target in zip(valid_x, valid_y):\n",
    "            x = x.reshape(1,-1)\n",
    "            pred = softmax(net.forward(x))\n",
    "            target = encode(target)\n",
    "            epoch_loss += log_loss(pred, target)\n",
    "        print(f\"Valid loss: {epoch_loss / len(valid_x)}\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
